90 comments

[ 2.7 ms ] story [ 141 ms ] thread
So that’s like a month of bitcoin transactions?
Two nodes in the Lightning network [1] could right now be performing 5M transactions per day and you'd be none the wiser (unless they'd tell you, that is).

And that's just 2 nodes out of the current 15K-20K.

[1] i.e. Bitcoin second layer.

Except that those transactions don't count until they get on the main chain and no one needs to send 1000 transactions of single penny, then have the exact same problems as everyone else when trying to get that to actually be usable.
> those transactions don't count

They don't? Then surely you wouldn't mind sending me a few bitcoins over a lightning channel? Pretty please?

> no one needs to send 1000 transactions of single penny

There is no such "single penny" limit.

> then have the exact same problems as everyone else

What problems?

I sent them over but I used a third layer network (a spreadsheet on my desktop)
(comment deleted)
(comment deleted)
Can't tell if sarcastic or not. But, that's perfectly legitimate. You could do that and if everything is signed correctly, the network will accept it.

I could write down a seed phrase on a piece of paper and hand it to you, and that's a completely offline transaction.

But just to be clear, unlike the hypothetical "spreadsheet on a desktop" (even with signed transactions), Lightning has more cryptographic assurances than that.

It prevents the parties of a transaction from double-spending balances behind each other's backs.

It also doesn't require trusting the other party once a transaction has (almost instantaneously) completed.

You don't need a network at all. Just email me who you want to transact with and I'll create a row for them.
Ok, so can I now redeem the BTC that you sent me earlier please, so that I can spend them in the first layer (or the second one)? Given that nobody else seems to want to use your third layer.

The fact that I even have to ask you already demonstrates how shitty your third layer is compared to Lightning, not to mention all the other obvious ways in which it's infinitely worse.

Typical Second Layerist. This is the future of money we're talking about, but you'd rather live in the past.
I know you're trolling and I even found your comments somewhat humorous, but I hope you're not misleading yourself (or others) into thinking that you have a valid point here or that your "third layer" scenario is in any way comparable to Lightning in anything except the most superficial ways...

In any case, seeing that I don't have a valid signed BTC transaction from you (because your 3rd layer suuucks), you don't want to allow free exchange between your layer and the upper layers, and I wouldn't want to bother with court litigation even if you were serious, well, have a nice day and good luck with your third layer lol

I actually said:

Except that those transactions don't count until they get on the main chain

While you quoted me as:

> those transactions don't count

You lost some of my message. Maybe your are reading it through a lightning node and it went offline half way through.

There is no such "single penny" limit.

No one said that, but lightning's examples are always these nonsense situations of paying a single party a thousand times.

What problems?

A transaction rate that could fit over the bandwidth of a 28.8 modem.

If you have to misquote people to make your point, maybe you're on the wrong side of the argument.

> You lost some of my message.

So what's your point? Your point is that the balance will be reflected in the first layer after some time instead of being reflected after each and every lightning transaction?

What does that have to do with what we were talking about? The point of this thread was how many transactions per time period you could do with Bitcoin, not whether the balance will be public information in real time?

> lightning's examples are always these nonsense situations of paying a single party a thousand times.

It's not a nonsense situation. Micropayments is something you can (trustlessly) do with Lightning but not with traditional payment technologies.

It doesn't mean you can only do micropayments with Lightning.

> A transaction rate that could fit over the bandwidth of a 28.8 modem.

I just told you that Lightning can transact orders of magnitude more than that.

You can do millions of Lightning transactions per Bitcoin transaction.

You are complaining about Bitcoin's transaction rate limit but my whole point is that Lightning allows you to have an effectively limitless transaction rate (under certain conditions, at least), even if it requires occasional Bitcoin transactions to settle balances.

Lightning allows you to have an effectively limitless transaction rate

Transactions that don't count and aren't useful to anyone until they become a bitcoin transaction, and that's still expensive and slow.

No one cares about it because it is terrible technology that doesn't actually solve anyone's problems, because the people that made it made the problems in the first place.

> Transactions that don't count and aren't useful to anyone until they become a bitcoin transaction,

I have no idea why you're making such an obviously false assertion.

These transactions obviously "count" and are useful to the entities running those 15K Lightning nodes (many on behalf of multiple customers), which can all transact amongst themselves over the network, even without touching the Bitcoin network (or occasionally doing it just to settle balances).

These transactions have almost exactly the same cryptographic and trustless assurances as normal bitcoin transactions, which are far superior assurances than those provided by traditional payment / money transfer technologies.

> No one cares about it because it is terrible technology that doesn't actually solve anyone's problems

I think you are confusing your own ideas and problems with other people's ideas and problems.

Just because you don't care about it doesn't mean that other people don't (or that even you won't in the future, if your circumstances change).

which can all transact amongst themselves

I can trade IOUs with a bunch of people in my local cult, but they aren't money until someone makes an actual USD transaction.

> I can trade IOUs with a bunch of people in my local cult

Lightning transactions are not based on IOUs.

IOUs can be defaulted upon, while BTC received over Lightning cannot.

Also, you need the cooperation of the other party to redeem IOUs. Lightning transactions don't require cooperation for settlement, they can be settled at any time by any of the parties.

All of this means that the BTC that is exchanged over Lightning is just as valuable as any other BTC, unlike IOUs.

> they aren't money until someone makes an actual USD transaction

That's because your definition of "money" is USD, so your sentence is a meaningless tautology.

It's like saying "raw beans aren't real food until someone cooks them". Uhm, sure, but does that mean raw beans aren't valuable or that you can't easily convert them into food if necessary?

And what's your point even? This point is equally applicable to the first layer.

BTC received over Lightning can easily be exchanged for USD if you want, or (even easier) for BTC in the first layer, so why does this point matter at all?

That's because your definition of "money" is USD, so your sentence is a meaningless tautology.

No, btc is money too, lightning transactions are useless until you sync them with the main network. Skip the middle man that no one wants. It has been 10 years, no one wants this, it doesn't solve anything. Every other currency has bigger blocks and never had this problem in the first place.

> lightning transactions are useless until you sync them with the main network.

This is absolutely false. Why do you even think this??

Heck, have you never seen people buying real world stuff with Lightning transactions in El Salvador?

Are you aware that if you receive BTC over lightning, you can spend them over lightning, with a different party, without hitting the main network?

> Skip the middle man that no one wants.

What middle man? Is the middle man in the room with us?

> no one wants this, it doesn't solve anything

False and false.

15K entities want this (at least, i.e. not counting hosted wallets). And it solves the limited transaction rate, which you were complaining about.

> Every other currency has bigger blocks and never had this problem in the first place.

And when they get moderately popular, the size of their block history becomes prohibitively expensive to download and verify.

All while not really fundamentally solving the problem, but rather only allowing a slightly faster transaction rate which would still not be able to compete with VISA (as an example).

What middle man? Is the middle man in the room with us?

The middle man is a layer that no one wants and it is not in the room with us now because no one wants it.

15k "Entities" means a bunch of VMs spun on the same servers.

And when they get moderately popular, the size of their block history becomes prohibitively expensive to download and verify.

This is where the technical rubber meets the road with all the lightning sycophants. This is insanely untrue and it never has been. No one is ever able to prove this with numbers, they get out the playbook of dealing with a claim they can't back up.

Bitcoin right now has the bandwidth of a 28.8 modem. At one point single transactions cost more than the space to store the entire chain. Normal people don't even need to sync with the main chain if they don't want to. The actual numbers of bandwidth and storage of this argument are so far removed from reality I can't believe people still say stuff like this. It's pure propaganda.

> Bitcoin right now has the bandwidth of a 28.8 modem

You're only considering the bandwidth to monitor/receive transaction confirmations in real-time and not the bandwidth to download the transaction history, which cumulatively grows over time, forever.

It's already at more than half a terabyte, which you certainly can't download over a 28.8 modem.

> At one point single transactions cost more than the space to store the entire chain.

The problem has never been about space, it has always been about bandwidth.

> Normal people don't even need to sync with the main chain if they don't want to.

Sure, but you need to put some bounds on how fast the main chain can grow to ensure that anyone (even the people, say, in El Salvador or the Central African Republic, both poor countries where Bitcoin is legal tender) can verify it if they want to.

> The actual numbers of bandwidth and storage of this argument are so far removed from reality I can't believe people still say stuff like this. It's pure propaganda.

Or perhaps you are privileged and you don't even notice, and therefore you are incapable of empathizing with others who are less privileged than you (in this specific instance)?

In any case, even if you were to ignore this, increasing the block size even by a factor of 100 would not fundamentally solve the problem.

It would still be a pathetically low transaction rate compared to Lightning and it would also certainly constrain Bitcoin usage as it was intended (as a trustless currency/asset).

That's not to say I would be against increasing the block size by a reasonably small factor, but I'm certainly not in a position to judge whether this should be done right now. I think it'd be better to ask the people in those countries, or even poorer ones who might one day make Bitcoin legal tender as well...

A $9 VPS with gigabit internet is literally one million times the bandwidth that bitcoin transactions take. ONE MILLION. Downloading the whole chain should take just over one hour. There is no argument here. You didn't use actual numbers at all. It's all fake propaganda nonsense. And again, someone using a cryptocurrency doesn't need to sync with the chain. Exchanges and nodes do.

Even though you already said it's all about bandwidth, next you'll say it's about the CPU to verify, even though that takes a second per block at most.

Then you'll say it's actually about hard drive space, even though a 512 GB SSD costs $24 dollars while fees for a single transaction spiked above $30 on average earlier this year. Right now they are at $6 USD!

https://bitinfocharts.com/comparison/bitcoin-transactionfees...

You didn't make a technical argument with numbers because you can't. Your next reply won't have real numbers either, it will be more attempts to obscure what you can't defend with attacks, distractions and more copy and pasted arguments that don't add up.

Worse, Lightning won't let you send pennies, so you have to send 10,000,000 transactions of 0.0000000001 BTC - each transaction consuming the equivalent energy of a big lightning bolt the size of a small lightning bolt (hence the name).
What the heck are you talking about?

None of what you just said makes any sense.

Pennies need converted to BTC first, which means many more transactions, which means that Lightning destroys the planet even faster than BTC. It's simple math.

BTC has been consuming all of the planets energy since at least 2020[1]. This increased transaction rate of Lightning just means that BTC will soon be consuming over twice the total energy produced by humanity, which is unsustainable.

Of course, people who don't understand the scientificness of this all are still defending bitcoin as if we don't already have enough payment apps.

  [1] https://www.weforum.org/agenda/2017/12/bitcoin-consume-more-power-than-world-2020/
I'm not a bitcoin zealot by any means but none of what you're saying is correct
> BTC has been consuming all of the planets energy since at least 2020[1].

That is plainly false[2], and the linked article does not say what your sentence purports it to say. :D

2: as evidence I can point out that I have seen at least one car moving since 2020. That was clearly using energy BTC did not consume. Same with observing at least one light bulb illuminating, or at least one person breathing.

Seems like more testing is in order:

> Upon the outage, both banks immediately activated IT disaster recovery and business continuity plans.

> "However," according to Tan, "both banks encountered technical issues which prevented them from fully recovering their affected systems at their respective backup datacenters – DBS due to a network misconfiguration and Citibank due to connectivity issues."

Perhaps regularly flipping between them on a regular basis, at least for a short time, so the 'passive' part of the active-passive pair gets some work, is worth considering.

"No recovery plan is worth anything if it is not tested regularly."

"An untested recovery plan is worth less than the paper on which it is printed."

I'm sure there are many other versions of the quote

But think of all the boxes you cannot check if you are honest about your recovery plan
If you haven’t tested your backup you don’t have one.
Have you seen this video of the guy who on some sort of “adventure” outing? They are way up in the air on some sort of platform where you are supposed to jump across these planks from one platform to another. Someone is filming from either a platform or plateau next to the rig.

There’s a harness connected to a rail above you to catch you if you fall.

In the video the man gets halfway across and the cable attached to his harness just pops off. The instructor behind him instinctively reaches out as if he can use The Force to catch him if he biffs, but he makes it to the other side unharmed, and is all smiles until the cable catches up and bops him in the back. At which point he turns around and connects the dots.

It’s a bit like that. And this is the image that comes into my mind every time I discover we have untested recovery processes while we are in the middle of a recovery.

"Backup--very, very boring. Restore--very, very exciting."
I’m surprised at this to be honest. At my employer (a bank) we have to do a flip a couple times a year and just continue running from the alternate DC until the next flip. Pain in the butt on the weekends of the flip but at least we know for sure that our DR plan works and is good enough to serve our regular load…
Probably observer's bias, but I've never seen a recovery plan work like it's supposed to.
I’ve worked a few places where the company did regular disaster recovery testing to ensure stuff worked how it was intended to. Most of the time, it worked. Every once in a while, it failed.
Yeah the "testing" worked....but did it really?

Maybe I'm just too skeptical.

Seeing this play out in IT makes me really worried when it comes to critical infrastructure.

On paper, there are plans and redundancies upon redundancies, and everything is perfectly safe.

In practice, I haven't seen them exercised, and some of these plans sound incredibly optimistic...

Humans are pretty good at scrambling around to hack something together when shit really hits the fan.
That scrambling often comes too late to avert serious damage or deaths though.
Or you could do chaos engineering: https://en.m.wikipedia.org/wiki/Chaos_engineering
I would venture to say that most non-tech startup companies are not equipped with the knowledge, the hardware, or the political fortitude to do something like that. Especially when it's customer's money at stake. It's one thing if Joe Schmo can't watch the latest Great British Baking Show. It's another matter entirely if Joe's paycheck is missing.
We used to do this for some large Artifactory clusters (~1bn artifacts) we ran on both the east and west coast of the US. Failing over to our DR cluster and our (internal) global DNS load balancer handled it just fine. Made upgrading those clusters seamless for the most part. Gave me confidence we could continue development in the event our main site fell into the ocean for some reason.
Bureaucracy triumphs over engineering considerations most of the time. Engineering division at banks might have suggested testing backup but top execs might disagree as it is unnecessary risk on operational system. Of-course they will play the blame game afterward and some low-level contractor will receive the heat cz of inability of decision making at top. Disclaimer: I worked as ELV contractor for an asset management megacorp.
Kind of a spicy opinion as a SRE, but if your disaster recovery strategy isn't part of your operational strategy then it's not actually a strategy. It's a moonshot in the face of failure.
That's a strategy. And depending on your downtime tolerance it may be an acceptable strategy. In this case it probably wasn't an appropriate strategy.
Sticking your face in front of a pit bull you’ve never met before is “a strategy”.

If you’re going to use the word strategy this way it has no meaning, which kills the conversation and communication in general.

I think you’re confusing strategy and behavior. Strategy is a plan to do better than default behavior.

Sticking your face in front of a pit bull is a strategy for what?

Writing up a plan for disaster recovery is a strategy for disaster recovery. The pros are that it is cheap and ticks the regulatory checkboxes. The downside is that it has a high probability of a high recovery time in the case of a disaster.

My point is that there are different pros and cons to every strategy that you can pick. I also don't think that all strategies are better than "default behaviour" whatever that is.

> Sticking your face in front of a pit bull is a strategy for what?

For meeting dogs of course.

The problem I have is that it seems like you agree with other people in this thread, but real world experience tells me that stating things this way is more fuel for the lazy and for anarchists than it is support for the SREs.

Don’t let the idiots think for a moment they are entitled to use a word as heavy as strategy. It ends farcically.

Writing it down doesn’t make it a strategy. Satire is also written down.

I think I can help disambiguate a bit. "is not a strategy" is a bit of a tip of the hat to "hope is not a strategy", the full extrapolation of which means that a plan is not equivalent to strategy and an unexercised strategy is the definition of hope.

An analog is when I was in the military we wore every level of MOP gear once a year to ensure that your gear functioned correctly and that you knew how to wear it. On a second day we would enter building filled with chemical irritant where you must dawn your mask, clear it, breath, and communicate. All of this is part of proving the plan which equivocates to strategy. We've not faced an imminent CBRN threat in decades, but we still do it because we maintain a strategy for dealing with it.

In this case, if you have a plan for how to deal with too much heat or too little power but you don't actively put the systems that transfer the responsibility or get put under stress during these disaster events then you never actually know for certain that they'll work. Therefore, they are not strategies, just plans.

In the business continuity/disaster recovery space we like to say, if you don't regularly test your DR plan, you don't really have one.
Moonshot is being generous.

It’s two steps away from Apollo 13. Build a CO2 scrubber out of parts in this box. You have twelve hours until the crew suffocates and dies.

> Perhaps regularly flipping between them on a regular basis [...] is worth considering.

True. But getting the PHB's at a bank (and perhaps the banking regulators) to sign off on doing that...could prove to be a non-trivial task.

It's odd that regular testing wasn't part of a compliance framework for them. BC/DR testing has been part of every security/compliance framework I've worked with.
Something something, the spare tire was flat.
> Question: Anything else besides multicultural tolerance that enabled Singapore's success?

> Answer: Air conditioning. Air conditioning was a most important invention for us, perhaps one of the signal inventions of history. It changed the nature of civilization by making development possible in the tropics.

https://www.vox.com/2015/3/23/8278085/singapore-lee-kuan-yew...

I'd believe it. Ever try doing desk work in a hot room? I get discomfort from sweating quickly.
(comment deleted)
The article seems to be talking about an outage that happened in Oct 2023, but then references a LinkedIn post that was written in 2021?

One of the banks, DBS, has had several outages in recent years, but this seems like sloppy writing.

Blaming contractors/subcontractors is so hot right now.

Cloudflare. Equinix. No accountability.

Does this surprise you? What is a corporation if not a legal entity designed to launder responsibility?

Have a problem with a service and you find out the company you bought from subcontracts to an LLC you've never heard of which in turn subcontracts to four other LLCs one of which is actually liable but only if you can figure out which one, and oh by the way yours has two employees and it's headquarters is a UPS Store box, good fucking luck.

> What is a corporation if not a legal entity designed to launder responsibility?

Corporations (and other legal entities used for business) are legal fictions that allow people to pool capital for productive endeavors and yes, with limited liability on the parts of the owners, because we as a society decided that encouraging productive endeavors this way is better than the alternative.

In a world with no limited liability, you could happily wring the owner's neck if they screwed up, but I'd imagine society as a whole would be a lot poorer off, because very few people would take that risk to engage in productive endeavors.

> Have a problem with a service and you find out the company you bought from subcontracts to an LLC you've never heard of which in turn subcontracts to four other LLCs one of which is actually liable

It doesn't matter, because the company you actually paid money to is the one that's responsible to you.

I'd imagine society as a whole would be a lot poorer off, because very few people would take that risk

Oh, people would make risky ventures alright, but you’d see a lot more private security of the Pinkerton variety.

>In a world with no limited liability, you could happily wring the owner's neck if they screwed up

I'd say more than a few people would like that opportunity with Boeing

> Corporations (and other legal entities used for business) are legal fictions that allow people to pool capital for productive endeavors and yes, with limited liability on the parts of the owners, because we as a society decided that encouraging productive endeavors this way is better than the alternative.

Worth noting that legal 'collectives' date back to Ancient Rome, city charters, monasteries, guilds in the Middle Ages, etc. While they are currently called corporation, from the Latin corpus for "body (of people)":

* https://en.wikipedia.org/wiki/Corporation#History

The original term was from the Latin universitas, and were universities in (relatively) more modern times when the concept saw uptick in usage (1100s AD).

* https://en.wikipedia.org/wiki/University#History

The first collectives were either groups of students coming together to hire teachers (Bologna), or teachers coming together (Paris).

* https://en.wikipedia.org/wiki/University_of_Bologna#History

* https://en.wikipedia.org/wiki/University_of_Paris#History

Your local municipality, county, country, etc, are all legal fictions that act as a legal person. Turns out the concept is handy in many situations:

* https://en.wikipedia.org/wiki/Legal_person

Considering how every vendor has been shoving cloud, serverless, and anything-but-on-premise solutions down everyone's throat for the better part of the century so far, does that really surprise you?

It's shitty landlord syndrome all over again. Tenants should have the right to complain and wave their sharpened pitchforks at them.

you get what you pay for, these companies have been racing to the bottom for years, trying to save that extra penny because IT is considered a cost center and always has been. In addition moving to the cloud is just a way to shrug off responsibility, it's not our fault it's AWS.....actually it is your fault because you were trying trying to save money and you handed off your business to the lowest priced contractor and now are surprised that the contractor doesn't care about your business nearly as much as you do.
> […] is so hot right now.

Literally in this case.

I wonder what the cost of a datacenter within Singapore is? The cost per sq/m in that country, which is the most expensive in the world, must be huge.
Well, considering you’d have most of the needed utilities already connected and present, a large population of well educated people, and extremely high bandwidth connections in close proximity, it’s probably only slightly higher than elsewhere.
I've run equipment in dc's in multiple places in the US, Germany, and Singapore. By far, the most expensive was Singapore.
Is it because it costs the operator a lot to run one or because they can charge a lot?

(I guess over time those should converge, but sometimes takes a while or doesn’t)

All of the above. Space/Power/Staff is just more expensive there. We also had a lot of trouble because a couple of our Dell servers had issues and it took some time to figure out how to get them repaired. There was just a lack of urgency there too.
Comparatively speaking, I would expect SG DC staff to be lower cost than western countries.

Power is the killer, and it's doubly bad. Not only is it expensive, you're using a lot of power to keep things cool given ambient temp fluctuates typically between 24-33C, 60%-90%Rh 24/7.

There was a moratorium on new DC builds in Singapore, so there's also a huge demand/premium on rack space in SG.

Singapore DCs build vertical to save on footprint. Quite a few DCs were built on the west side of the island where it's industrial and land is/was cheaper.
The overheating datacenter didn't stop the transactions. It was the technical inability of the banks to fail over which should have been seamless.

Outages happen.

IMHO, "fail over" will always fail. Any multi-data-center architecture should actively utilize all data centers, all the time, maybe dedicate more traffic to one or the other. But, if you've got a "passive" data center that you wanna activate during an outage, you're gonna have a bad day. Every time.
My experience contradicts your hyperbole. Over the past two decades I've been part of several fail overs with active/passive and build from scratch DR solutions. Most succeeded. The ones that didn't succeed on the first attempt did so on the second one. Not a perfect track record, but it does happen with proper documentation, testing, and skilled staff to handle it.
How to make this problem even worse: make all the banks in the street use the same datacenters, namely either AWS or Azure.
AWS version "Overheating employee stopped 2.5M bank transacti..."
In the EU, the Digital Operational Resilience Act (DORA) mandates that financial services companies must manage their cloud risk - https://www.digital-operational-resilience-act.com/

Could lead to more multi-cloud adoption among financial services companies who are in scope.

(no relation to the other DORA, DevOps Research and Assessment, by the way)

Crypto does not overheat.
We are no longer in an era where this should be allowed to happen, even for say... a legacy application that runs on Windows 2000 and needs NTFS on a network block device.

After 9/11, the "zero nines" whitepaper was released to push for cloud (never let a disaster go to waste).

I have hands-on deployed many K8s+Ceph clusters. Ceph's block storage can be deployed multi-AZ where multiple datacenters each with their own power systems are in close proximity. There are methods like Raft (used by K8s and Patroni) and a really great method demonstrated by Google's CloudSQL that uses a SQL table to infer which SQL server should be a "leader".

So like wtf guys.

Of course they blame the mistake of the single contractor rather than their own inadequate system monitoring.
(comment deleted)