This was a claim (according to the article) of specific group who goes by the name New World Hackers.
The word hackers has very wide meaning and this title can be considered to be insulting to multiple subgroups who like to think of them as hackers in a good sense.
"...attacks were merely a test, and claimed that the next target will be the Russian government for committing alleged cyberattacks against the U.S. earlier this year."
I think its just the good old 1990s scenario repeating itself, only this time with IoT, rather than usual IT.
First, a mad rush towards the IoT where every startup and their dog wants to setup their business on IoT. Then comes fear as the hackers try to exploit every hole in IoT devices. And finally, once people realize the importance of keeping with the software upgrades, the market will settle somewhere in between.
The thing is there will be no upgrades for the IoT. Just look at all the Android phones abandoned by their manufacturers. Do you think Phillips is going to pay someone to produce patches for a 5 year old light bulb, or smoke detector, or whatever?
If it becomes to bad and people start to pay attention, yeah. If there were more malware doing (edit: very visible) stupid shit around, people would learn quite quickly (instead of malware that tries to hide and do things that don't effect the device owner), this way we'll see where it goes.
Most compromises in things like this are going to be invisible almost by accident. Unless your light bulb is maxing out your upload for extended periods of time, or the police come knocking on your door when your light bulb does something very illegal and very loud, nobody is ever going to notice.
I meant "stupid shit" that's intended to be noticed. The equivalent to malware of olden days that would intentionally bluescreen your machine, display a stupid image or something. Brick devices, turn all lightbulbs into disco mode, replace video camera images with static or porn, de-auth all WLAN devices around...
But nowadays malware is less about "hijinks" and more about criminal enterprises that profit from staying hidden, so pressure on device owners and makers is going to have to come from somewhere else: ISPs, governments, ...
I wonder why they test DDoSed US and EU infrastructure if they were planning to target Russia from the beginning. This is plain silly and suggests script kiddies are involved. Or someone claiming to be the perpetrators spreading FUD.
> I bet that a greyhat will find a way to brick a lot of these cameras.
A chaos monkey for internet security - now that's an interesting thought!
That could actually work. Market forces aren't strong enough currently to force manufacturers to get security right, since the average user doesn't really care that much about it. But if it became common for any poorly secured product to turn into a brick as soon as you connect it to the Internet, users would pretty quickly figure out what brands to buy and bring some serious pressure to bear on the others...
Wow. I've done research on smart-locks and their poor security models before, but this is news to me. I hear people bemoaning introducing govt. regulation, but I honestly cannot see another way out of this; these manufacturers clearly do not care!
Just bring it, hackers! Drop 100Tbps on every critical service to show them what's up! Hit all of them with everything you got!
I've been waiting for a good illustration to regulators of why POTS, leased lines, and satellite must continue to get investment & kept separate. On top of the small cases I have. All these people's shit is going down whereas the people using dial-up to BBS's or leased lines between critical sites are still doing just fine. I mean, they do gripe about speed or pricing more than the rest of us but that's the kind of problems they're used to living with. ;)
Note: The major attacks will also get regulators' attention to IoT risk. I have recommendations of techniques and existing products ready for that, too.
Note 2: I think someone will see this saying, "OMG! He's encouraging crimes and damage to happen! WHY!?" Actually, apathy by consumers and suppliers is why it happens and will get worse regardless of what I say. I'm just waiting for the inevitable as an opportunity to improve the situation.
"Due to increased DDOS attacks, the U.S. and European nations' governments felt their hand was forced. Swift and sweeping legislation was passed to 'keep you safe' online, including deep tracking and identification of every activity taken online. The number of personnel hires for citizen monitoring increased tenfold in the pursuit of catching hackers, in a new global program that governments are calling, 'no stone unturned'.
Private contractors were swift to step in with their offer, providing ever more efficient and penetrating tools to get this job done. Citizens never felt so safe!"
It's kinda silly to make a dry run like that, because people in charge of the attacked systems will harden them and the attack is much less likely to work next time.
Not at all the case with IoT. See Bruce Schneier's recent article on the matter, where he makes the case that government regulation is the only feasible remedy at this point, as "The market can't fix this because neither the buyer nor the seller cares." [1].
I meant the servers being attacked. The owners of them definitely care.
As for the IoT, the appliances could have their firmware in ROM instead of flash. Then, the malmare would not survive a reboot. Many customers are large scale enough (Microsoft, Google, governments, etc.) that they can demand it of vendors and vendors will deliver. Really, how often do you desire to update the firmware on your hard drive, your USB stick, etc.?
(Another way to do it is to have the write-enable line controlled by a physical switch or jumper.)
The only thing I can figure is everyone has forgotten what ROM is.
If there's a ROM with non-editable software it will just get instantly compromised as soon as it comes back up. For your standard "internet of things" device there is no room, physically or in the bill of materials, for things like connectors for people to physically deliver updates.
This is not a difficult problem to solve. The ROM cannot be overwritten - hence it can be designed so that malware cannot run.
Also, jumpers are cheap. Just set the jumper to enable writes, and download the update.
Are you happy with it being unknowable which of your appliances are compromised or not? Would you pay $1 more for a disk drive with firmware in ROM? I would. If you were running a banking system, would you pay extra for code in ROM that cannot be compromised?
You would. But you're not the market - 99.99% of other people, who don't even know what a "jumper" is, are the market. So your preferences don't matter. Yes, even in most technology products.
As for IoT devices, I can't imagine average Joe or Jane prying off their smart whatever, destroying the pretty plastic casing it's hidden in, and manually setting jumpers to flash firmware.
Isn't it ironic that the means of updating firmware to prevent installation of malware is the vector for installing malware?
I don't think the current scheme is working very well. Even worse, there is no way to tell if your appliance has been compromised or not. There are a lot of companies that care whether their machines are infected or not.
What happens when you can't update the IoT device then, reboot may remove the attacker but the IP will be the same (and even if it changes, probably wouldn't be long before the next address is scanned and put back into being a zombie) and they would be right back in if you can't update and patch the flaw that let them in.
I think in all honesty the regulation sounds best, requiring the router makers to disable features such as UPNP would prevent massive attacks like these.
Why? The rest of your comment doesn't address that.
>See Bruce Schneier's recent article on the matter, where he makes the case that government regulation is the only feasible remedy at this point
He also doesn't explain why. He literally writes:
>>In fact, the software used to attack Krebs was simple and amateurish. What this attack demonstrates is that the economics of the IoT mean that it will remain insecure unless government steps in to fix the problem. This is a market failure that can't get fixed on its own.
Why does it demonstrate that? "Neither the seller nor the buyer care" can also be applied to traditional botnets using computers - if the botnet is clever enough to not make any noise. He "justifies" the "sellers don't care" part with:
>>they're now selling newer and better models, and the original buyers only cared about price and features.
... again the same for smartphones or laptops. And furthermore, what regulation he proposes to solve this? I mean, he presents the issue as being:
>>The security of our computers and phones also comes from the fact that we replace them regularly. We buy new laptops every few years. We get new phones even more frequently. This isn't true for all of the embedded IoT systems. They last for years, even decades. We might buy a new DVR every five or ten years. We replace our refrigerator every 25 years. We replace our thermostat approximately never.
Would he suggest that the companies selling these devices be obliged to push security patches? The same companies that he wrote could not even afford testing?:
>>Companies like Microsoft, Apple, and Google spend a lot of time testing their code before it's released, and quickly patch vulnerabilities when they're discovered. Those companies can support such teams because those companies make a huge amount of money, either directly or indirectly, from their software -- and, in part, compete on its security. This isn't true of embedded systems like digital video recorders or home routers.
...
I have read "government is the only solution" at least 6 times in that short article, without exaggeration. Looks a lot like fearmongering and propaganda to me.
>> In fact, the software used to attack Krebs was simple and amateurish. What this attack demonstrates is that the economics of the IoT mean that it will remain insecure unless government steps in to fix the problem. This is a market failure that can't get fixed on its own.
> Why does it demonstrate that? "Neither the seller nor the buyer care" can also be applied to traditional botnets using computers - if the botnet is clever enough to not make any noise. He "justifies" the "sellers don't care" part with:
Not really. When a desktop PC is compromised sometimes it is noticed because eventually malware comes along that bricks the system, installs ransomware, or otherwise discloses itself.
But the sad truth is that a lot of compromised desktop PCs aren't noticed which is why there are millions used in botnets right now. Microsoft decided to take it seriously and pushed automatic updates, adopted secure coding standards, etc. It took a decade to make a good dent in the problem and it still isn't completely resolved.
>>they're now selling newer and better models, and the original buyers only cared about price and features.
>... again the same for smartphones or laptops. And furthermore, what regulation he proposes to solve this? I mean, he presents the issue as being:
>>The security of our computers and phones also comes from the fact that we replace them regularly. We buy new laptops every few years. We get new phones even more frequently. This isn't true for all of the embedded IoT systems. They last for years, even decades. We might buy a new DVR every five or ten years. We replace our refrigerator every 25 years. We replace our thermostat approximately never.
>Would he suggest that the companies selling these devices be obliged to push security patches? The same companies that he wrote could not even afford testing?:
If all manufacturers have to meet certain standards then no one can undercut anyone else by skipping out. You can save some money by not including various safety features but then you can't sell those devices to over half the world so what's the point? If it is all voluntary you have busy consumers who don't have the requisite background knowledge (nor the time) to learn the details of software security, nor audit the code for secure coding standards and that's assuming the code was even available. If you want to make the claim that every human being should be an expert at evaluating such claims that's a massive inefficiency.
Really that's the primary problem with libertarian ideas in general: it introduces massive inefficiencies in the market. Not everyone can be an expert in chemistry, software, mechanical design, food safety, et al simultaneously. (Not to mention needing perfect and instantaneous information). Claiming that lawsuits will fix everything after-the-fact ignores the real human cost, the lost potential of the ruined lives, and doesn't work unless you completely eliminate the corporate veil and make everyone personally liable. Otherwise every company will just be incorporated with various shells to ensure you can't actually recover anything. That goes on today - get sued? File bankruptcy and re-incorporate under a new name. You don't even need to move the factory because the seller was just leasing it from your shell incorporated in the Bahamas.
>>Companies like Microsoft, Apple, and Google spend a lot of time testing their code before it's released, and quickly patch vulnerabilities when they're discovered. Those companies can support such teams because those companies make a huge amount of money, either directly or indirectly, from their software -- and, in part, compete on its security. This isn't true of embedded systems like digital video recorders or home routers.
...
>I have read "government is the only solution" at least 6 times in that short article, without exaggeration. Looks a lot like fearmongering and...
If said companies cannot afford to test or provide patches, maybe they shouldn't be in the game of creating unpatchable internet-facing devices.
IoT companies have been pushing crappy proprietary solutions for a while now; mobile phones and computers are exempt from this primarily because they run a few major OSes, which can be relatively easily patched. IoT devices are a shithole of poorly written code that doesn't even work properly, let alone securely.
I'd prefer it that way, to be honest - because your average small company has exactly zero incentive to deliver actually useful product. They can easily sell crap and avoid consequences. It's how software is done too, except that a crappy SaaS startup can't be easily turned into a botnet.
Not that big companies don't try to sell crap - but they can't so easily evade consequences, so they at least have to think about self-preservation.
Couldn't they create a completely separate IOT protocol, so only connections expecting to receive connections from that protocol would receive it--so essentially a IOC device could only ping specific IP's that use a different spec like 255.255.255.255.255 or something.. Essentially a separate internet that can't mess up networks on the main internet? It could still use the internet as a gateway, but would give ddos' protection more ability to block things.
Ask anyone involved in rolling out IPv6 how painful it is to get every middlebox on the public Internet to route a different protocol besides the original IPv4.
What about a special set of bytes that are standardized that all IOT machines should send in every request. Some sort of IS_IOT signature. That way if there's a huge influx, all messages containing the signature could be filtered.
1. Introduction
Firewalls [CBR03], packet filters, intrusion detection systems, and
the like often have difficulty distinguishing between packets that
have malicious intent and those that are merely unusual. The problem
is that making such determinations is hard. To solve this problem,
we define a security flag, known as the "evil" bit, in the IPv4
[RFC791] header. Benign packets have this bit set to 0; those that
are used for an attack will have the bit set to 1.
Quite correct, but nonetheless, that RFC does explore the topic of "why not just mark the traffic?" One basic objection which detaro pointed out is that there's nothing to stop attackers from clearing such marks.
As Wikipedia puts it:
> The evil bit has become a synonym for all attempts to seek simple technical solutions for difficult human social problems which require the willing participation of malicious actors
Ok so I don't know that much about how DDoS attacks work besides that it's basically a large influx of input that makes it impossible for 'legitimate' users to get through.
But in a block chain, a Sybil attack—imagine voter election fraud, where numerous fake IDs are made to vote for a candidate—is blocked by making the cost of generating hash values (government ID is pretty hard to counterfeit) extremely high.
Can the same methodology be applied to blocked DDoS attacks?
Like using Dyn to find a domain is similar to making a phone booth call to yellow book call center, why not just raise the price of calling from 25c to $1.00?
Because the whole internet would become unusable if DNS lookups cost you a dollar.
In the first place, the requests are being made by owned IoT devices. So making the requests more expensive (in time, bandwidth, or actual money) would just hurt the owners of those devices. The people operating the botnet wouldn't care, they'd just get a larger botnet.
I don't mean an actual dollar cost but computing power.
Is there a way to enforce a limit on requests from IoT, or a way to increase costs of running a botnet but not have that cost transfer to the owners of IoT?
IoT botnets are a problem because they're massively distributed. Even if a single device could only issue one request per second you'd just need to own more devices. There are millions and millions of vulnerable boxes out there, sometimes dozens in one home or office.
Things get crazier once you have one compromised hole-punched device that is behind your NAT/firewall too. You can end up in a situation where you have a local C2 server on your light bulb, controlling a sub-botnet of your fridge, washing machine, and wifi controlled doorbell. It's not unreasonable to think that this could compromise orders of magnitude more devices than the original botnet itself, considering how many things are just saved by their interface not being routable or accessible from the wider internet.
Something I've wondered is how much of an impact large botnets have on other systems they rely on. If you had a million compromised CPUs in a small geographic area suddenly jump from a low power state into doing a massive amount of work, could it cause localized brown outs? Some napkin math says probably not, but it's not something I've heard considered much before.
A few hundred megawatts is well within the amount of power peaker plants can produce and the vast majority are idle >> 90% of the time. You'd need millions of air conditioners to cause a brownout. Most states fine utilities heavily for any disruption of service (to the tune of $10s or 100s of millions per company per incident) in exchange for a govt granted monopoly so they all have extensive capacity built out, capable of providing for huge extended spikes in consumption.
Imagine if all those IP cameras, routers, NAT boxes and what-have-you had been designed with one simple policy: the internet port doesn't work until the user sets a password.
Even very lame passwords might be expected to reduce the effectiveness of this attack approach by an order of magnitude or two.
I was just saying to a friend yesterday that this would be a great policy. I think it would go a long way.
But the problem remains that these devices, more often than not just don't get updated. So in a year or two, there will probably be a handful of exploitable issues that won't ever get patched...
How many Windows users are using a pirated version of their operating system that gets no updates? How many of those using the genuine version ignore/disabled it? How many update their anti-virus? Custom ROM users? How many Ubuntu users ignore the updates? ...
Because it breaks things. Updates sometimes break things and require human intervention to get the device running properly again. This can happen since systems use different hardware. With IoT it's an issue to a lesser extent, but you can also brick those with an upgrade.
Better an individual device has some trivial breakage for a short period of time than the whole Internet breaks. I don't think we're there yet, but it feels like we'll need to take some pretty draconian steps if we continue the way we're going. Doesn't it make sense for 99% of IoT devices to only be able to communicate via a router over WiFi?
We are talking about common desktop PCs with more or less standard set of packages, not about server machines. I seriously doubt there are any breakages there.
I've noticed that sometimes updates to Firefox cause it to stop working until it's restarted. If you're writing a long message, or filling in some form, or if you have an order confirmation page open, then you might lose that data.
I've never seen that happen in native GNU/Linux applications, though. I guess it might be related to the XUL stuff in Firefox.
But given that Firefox (and Chrome/ium) are memory hogs that eventually get OOM killed, you need to restart them periodically anyway, so it might not make a big difference if the OS does auto-updates in the background.
The windows 10 anniversary update was a nightmare for me, my office, and seemingly a lot of people[0]. Note that the update installed itself too, opting to alert me at 7pm one evening while my machine was running unattended that it was going to restart in 15 minutes.
The point of my comment was that, yes, this kind of policy would be good and might have stopped this attack.
But it wouldn't solve the whole problem, just make it a tiny bit more difficult for the attackers. Definitely worth doing, but we will need to address the culture of 'ship and forget' also or we will just have the same problem but with software exploits.
If routers weren't such a crapshot security wise as well, they'd be a great integration point.
Most better routers nowadays can already isolate devices from the internet/whitelist specific domains (family filter functions) and offer VPNs from the outside. From a technology perspective that's most of the pieces you need. Make a better UPnP implementation with user confirmation, make it easier to configure and everybody can get their devices nicely isolated-but-accessible.
Most modem/routers that ISPs give you come with a password that is defaulted to something random - different for every device - and a sticker with that password somewhere on the physical device.
That seems like the best solution. Now we just need to enforce that as a standard with internet connected devices.
After a period of time of coming out of the box and set to open, followed by predictable passwords, followed by a lack of rate limiting on WPS and vulnerable TR-06 setups and the always present mistakes in router firmware allowing remote exploits.
Now if only we could get a standard router manufactures to stick to :-P
Sadly I can see IoT having to go though the same slow learning curve.
Haha, yes. Except when the password isn't random. The biggest ISP in Norway delivered multimodems with seemingly random SSID and password. That worked fine, until someone figured out that the password were derived from the SSID with an algorithm (Or something along those tracks, I can't remember the specifics). Now we had thousands upon thousands of basically free wifi hotspots!
You can imagine why that's a non-starter. You'd be locking millions of people out of their Internet access with no workable way for them to get it back.
Just imagine how many scammers would pop up to 'secure' things for people locked offline.
No it shifts our priorities. Once people realize they can't access 90%of the internet unless they reinstall their OS or unplug all network devices, people will become more vigilent.
It's understood that the recent DDOS attack on Dyn via the Mirai bonet was entirely made of up IoT devices with just 62 default passwords https://news.ycombinator.com/item?id=12766950
Or ship every new device with a different randomly generated 10+ character password and writing it on the device's label just like the MAC address. I agree you would need physical access to the device to gain access to it but you can always change the password to avoid this. There are vendors that already do this with the WiFi key.
This would be a nice improvement, but as others point out, more flaws will exist and be exploited.
Really this is about incentives: manufactures and users have little incentive to worry about security if the losers are third-party the targets of a DDOS.
Things will only improve when there is liability: either the manufacturers or the users get fined when their stuff becomes an attack vector. Fining users will probably never be politically possible, but I suspect it would be the better option.
I think fining manufactures would lead to scenarios where we fight the last war, much like how the 2008 financial crisis was not prevented by the existing nest of regulations which were passed to prevent the previous crises.
That is fining manufacturers would lead to strict and expensive regulations saying just how things must be done, but which are laughed at by baddies developing new exploits that work within these particular rules.
On the other hand, if the rule is "if your box does bad stuff, then you must pay" -- then consumers will look to security as part of the good reputation of the vendor.
That extra layer of indirection is what I look for to avoid "last war" type scenarios.
When is someone going to make a robin hood malware that helps resolve these issues and fights off other malicious tools trying to use these insecure devices?
Regardless of any good intentions, it'd be a pretty grey area to deploy a tool that cleaned and then protected against future attacks on a device. The exploit (well, at least the one currently drawing attention) is targeted at devices running BusyBox that are compromised over telnet with a embarrassingly small dictionary attack. If you could clean any existing instances, the protection for future exploits is simple - set a randomly generated password.
The downside is doing this (without approval) to remote devices would likely break device functionality. The majority of users/owners of these devices are likely completely unaware they have been compromised today as it has little to no effect on anything they can see. Sure, a robing hood malware could clean up this mess pretty effectively, but the little side effect of stopping devices from, you know, working is probably undesirable.
It would be almost useless since localhost doesn't even generate outside traffic. It will only drive up the electricity consumption and heating the device. One could run a fork bomb and temporarily brick or permanently damage the device (due to overheating) or disable networking altogether.
Ignorant thinking out loud here. Is it feasible for certain infrastructure providers to team up and collect the ip addresses of the requests. This gives us a list of IPs with bots. THEN google, Facebook, Twitter, etc (major web properties) use this list to notify any of its users that one or more of their devices have been compromised and point them to a how to guide for securing it. Or more simple a script to patch it.
My thinking is that the only way to stop this is to get users to lock down their insecure devices.
A constant banner would annoy most people to action. Especially if it was really easy to issue a fix. Download and run. Script determines the actual device causing the harm and patches it. Possibly asking for a new password from user. I guess this assumes that a script could be written to issue a patch for the majority of cases. If not, then how is this device connected? Script could detect router and apply fix at that point.
So we need:
1. IP addresses for the sources of the attack load.
2. A way to distribute patches for those IP addresses.
I propose:
1. Those suffering the attacks can provide this.
2. Notify users via websites coordinating on displaying an alert with tools to patch.
Alternative solution to #2:
Can some part of the ISPs that connect these IPs be patched to detect and block malicious attacks?
Back in the day, the Internet was far simpler, with far less bullshit. If your IP was spewing crap, you would hear about it. And you either resolved the problem, or got null-routed.
The Internet is far^N larger and more complex, for sure. But accountability still has a place, I think.
I wonder if Dyn will at least release a heat map of source IPs.
I suppose that points to the solution needing to be at the ISPs. They know the IP assignment history. And they could also be responsible for filtering malicious traffic.
Interesting I didn't know this. ISPs could certainly change this policy for a duration during which they scanned their networks for Mirai IoT devices and notified affected customers.
In the US I believe cable modems are static DHCP leases tied to the mac address of the cable modem.
Of course this would be an arms race of sorts if the next version of the malware simply added port knocking sequences to the compromised routers in front of these vulnerable IoT devices.
Or, ISPs could simply block all service to offending IP addresses. The customer would see a notice: "The following IP address delivered N Gb of traffic over our network as part of a malicious DDOS attack. Please patch or isolate the offending device and contact customer service at 123-456-7890 to restore service."
I guess that wouldn't make your customers super happy, but it would probably be effective.
> If every service provider did the same it wouldn't affect retention.
If that would somehow happen, you could use this miracle to solve most of the problems on this planet. This is literally the biggest issue of humanity - it's extremely hard to get people to coordinate. Doing so usually requires forcing various incentives to align.
In this case, you'd most likely need a direct government involvement to make all ISPs behave.
That's exactly what government regulation addresses. A truly global approach would be nigh-on impossible, but it should be possible to get all ISPs in the states to come on board, for example.
Despite this, I consider it very likely in probably the next month or two that were going to see ISPs blocking people for this. All the objections as to why it's hard are true, but ISPs can't do nothing, either... a DDoSed internet generates angry customers and support calls too, along with everybody else being angry. And nobody else can do anything about it either; it's the best, if not the only, choice in the set of bad choices we now face.
I'd be surprised if they aren't working on it right now.
This is an attack over UDP. The source address is likely spoofed. Simply blocking an IP will do nothing but banning somebody not related to the attack at all.
I don't think you understand how spoofing works. And no, this attack was not spoofed. It was just hundreds of thousands of IoT devices each pushing a small amount of traffic (0.5-1mbps).
I don't believe this Mirai IoT attack is using spoofing. But on the topic of spoofing: if an ISP doesn't follow BCP38 and allows spoofed packets to leave their network, then there is truly no traceability in where a packet came from. See this talk from Strangeloop about how spoofed packets are a problem [0]
I think there's might be a significant legal issue to an ISP patching customers devices. Also a spoofed IP renders the source meaningless.
">THEN google, Facebook, Twitter, etc (major web properties) use this list to notify any of its users that one or more of their devices have been compromised and point them to a how to guide for securing it."
I am not sure I want FB and Google policing the internet. They have enough power as it is. This would set a bad precedent.
I have the same question. Countless articles explain these attacks are being carried out via IoT devices but I've seen no mention of specific brands/models. Are there worst offenders? Are common devices like Nest, Wink, etc vulnerable?
> "I think these statements are bogus. Russian government is a much smaller target than Twitter."
It's not about the size of the infrastructure, it's about the size of the political statement. Targeting larger infrastructure first appears to have been a move to demonstrate the power of their botnet.
My experience from talking with non-techies suggests that they'll never understand the problem is the lack of security in their devices & network. All they do is act victim and get enraged about hackers. The thought that they should demand security from the companies who sold them the vulnerable devices doesn't cross their mind, and if try to sell them such an idea, they will protest and call for justice on the principle that it is the hackers who are being criminal and it's not the victim's or his devices' responsibility to keep themselves safe. Just as they don't expect to have to live in a bunker with inpenetrable locks to stay reasonably safe from burglary.
Stretching a bit the burglary analogy you could put it like: "imagine a guy selling you a door lock that can be opened by anyone and then you use it to lock a shootgun, can you shift all the blame to the guy who just took it an shoot your neighbor?"
Many locks in real life are symbolic. Interior doors for instance are hollow and can be breached with no trouble. They serve mostly to keep the law-abiding from snooping casually, and to provide evidence of theft (forced entry).
A pity we can't have digital locks that 'break' when used, leaving some fingerprint from the perpretrator.
Yeah? Then privision a new box and try to ssh to it with port 22 blocked. I could of course run fail2ban or a firewall rule in order to block at 3 failed attempts to connect and still run on port 22. Or use port knocking.
Or just use an alternate port. There are, what, 65533 alternate ports available?
> Then privision a new box and try to ssh to it with port 22 blocked.
Shouldn't be a problem if you've got your provisioning stuff set to configure ssh on an alternate port.
> I could of course run fail2ban or a firewall rule in order to block at 3 failed attempts
A good idea, but still an additional step beyond simply using a different port.
> and still run on port 22
And you'll still end up on the receiving end of more malicious traffic than if you'd not used port 22.
> Or use port knocking.
Not necessarily a bad idea, but still far more complex than simply using a non-default port. Same reason that many ISPs (including Amazon) place more restrictions on port 25 than other ports commonly used for SMTP traffic.
- If you don't have a very well defined need for a short DNS time to live, set your time to live to a large value, perhaps a day. Then, as long as someone can get at least one DNS request through, they can reach your site all day. (Ycombinator.com, why do you have a TTL of 12 seconds?.)
- Get multiple DNS services now. Not just two. Get four or five, some of which are not widely used.
Those two things will probably get you through future DNS attacks.
Can anyone comment on the best way to run multiple DNS services?
Is this as simple as setting up the same records on multiple providers and updating your nameservers to point to the different providers? Or is there more involved?
Are there any providers which will replicate records from your 'master' provider, or is this going to be manual?
To put it simply, yes. You have multiple NS records, and each points to a server that can act as an authoritative name server for your zone. When clients query for your zone from parent zones, they'll get all your name servers. The hard part is how you keep those name servers synchronised.
IIRC, bind has multiple built-in methods of keeping zone files synchronised between boxes.
We use BuddyNS (https://www.buddyns.com/) as our secondary DNS provider. They use AXFR to automatically sync their servers with your primary DNS servers.
You just need a primary DNS vendor which supports AXFR, such as DNS Park (https://www.dnspark.com/).
No. My domain uses five DNS servers under three different sysadmins. I can't
possibly expect that fellow admins that were kind enough to run for me backup
NS-es give me shell access to their systems.
It doesn't help if the DNS provider has multiple servers, and they're being attacked. You need unrelated DNS providers, and not the same unrelated DNS servers others are using.
I've always thought about expanding on this and just keeping a constantly-updating cache of any DNS requests I make (Where the system keeps them and serves them even after expiry if not updateable).
The only problem is this seems be a tough one to get right config-wise and I just haven't had time to figure it out.
Anybody know of any good tutorials for this specific use case? I've googled it a few times and they never seem too easy.
> Anybody know of any good tutorials for this specific use case? I've googled it a few times and they never seem too easy.
While it's not exactly what you're looking for, you could use dnsmasq, you could use the "--min-cache-ttl" option to increase the TTL: but only up to an hour. You can patch the source to increase this time to suit your needs.
To exactly match your use-case dsnmasq would need to have a regular cache that respects upstream TTLs and a last-resort cache that doesn't.
> Ycombinator.com, why do you have a TTL of 12 seconds?
You either meant to say 60 seconds, or forgot to factor your local caching DNS resolver in. Here:
$ dig +short soa ycombinator.com
ns-225.awsdns-28.com. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400
$ dig a ycombinator.com @ns-225.awsdns-28.com. | grep -A3 'ANSWER SECTION'
;; ANSWER SECTION:
ycombinator.com. 60 IN A 54.240.184.14
ycombinator.com. 60 IN A 54.240.184.154
ycombinator.com. 60 IN A 54.240.184.206
CDN providers has a highly volatile infrastructure setup, short TTL helps them to dynamically reallocate resources. Just as an example, cache invalidation could be implemented with a switch of DNS records.
>"CDN providers has a highly volatile infrastructure setup"
No they don't. CDNs with the exception of AWS Cloudfront don't run on a cloud provider. Things aren't quickly spun up and down with any kind of velocity. CDNs manage all of their own infrastructure, POPs, hardware etc. There is usually a maximal hardware lifecycle since the CDN business is so CapEx heavy.
Cache-Control TTLs and DNS SOA record TTLs have nothing to do with each other. They are separate layer 7 concerns. The former is used by your browser and the latter by your system's resolver library.
I have never heard of anybody using SOA TTLs records to purge a cache. How would that even work?
CDNs generally give you decent tools to purge your objects.
Sorry to chime in late, a day after this comment was added.
If you set a long DNS TTL, then you will not be able to move your IP addresses quick enough. In the DDoS world, you may need to nullroute an HTTP server IP. If you do that, you assume you can point the DNS to another, not-nulled IP. For this reason, keeping the DNS TTL to low values will make you more resilient.
Multiple DNS providers is probably a good advice, although hard to do in practice. The api's of different vendors don't always match. (cname flattening? geo-routing?). Also a real big attack will just likely take down all your providers. Using multiple providers may well _decrease_ the stability of the internet at large.
DDoS against DNS is not trivial problem. As sad as it may be, I'd say we must just assume DNS infrastructure in the world works. If it stops, then, many users will be affected.
204 comments
[ 2.0 ms ] story [ 211 ms ] threadYou mean they don't qualify?
The word hackers has very wide meaning and this title can be considered to be insulting to multiple subgroups who like to think of them as hackers in a good sense.
Worth noting:
>we are in Russia
Also their website default page looks like DDoS UI.
"...attacks were merely a test, and claimed that the next target will be the Russian government for committing alleged cyberattacks against the U.S. earlier this year."
First, a mad rush towards the IoT where every startup and their dog wants to setup their business on IoT. Then comes fear as the hackers try to exploit every hole in IoT devices. And finally, once people realize the importance of keeping with the software upgrades, the market will settle somewhere in between.
But nowadays malware is less about "hijinks" and more about criminal enterprises that profit from staying hidden, so pressure on device owners and makers is going to have to come from somewhere else: ISPs, governments, ...
I guess we'll know soon.
I bet that a greyhat will find a way to brick a lot of these cameras.
A chaos monkey for internet security - now that's an interesting thought!
That could actually work. Market forces aren't strong enough currently to force manufacturers to get security right, since the average user doesn't really care that much about it. But if it became common for any poorly secured product to turn into a brick as soon as you connect it to the Internet, users would pretty quickly figure out what brands to buy and bring some serious pressure to bear on the others...
I've been waiting for a good illustration to regulators of why POTS, leased lines, and satellite must continue to get investment & kept separate. On top of the small cases I have. All these people's shit is going down whereas the people using dial-up to BBS's or leased lines between critical sites are still doing just fine. I mean, they do gripe about speed or pricing more than the rest of us but that's the kind of problems they're used to living with. ;)
Note: The major attacks will also get regulators' attention to IoT risk. I have recommendations of techniques and existing products ready for that, too.
Note 2: I think someone will see this saying, "OMG! He's encouraging crimes and damage to happen! WHY!?" Actually, apathy by consumers and suppliers is why it happens and will get worse regardless of what I say. I'm just waiting for the inevitable as an opportunity to improve the situation.
Private contractors were swift to step in with their offer, providing ever more efficient and penetrating tools to get this job done. Citizens never felt so safe!"
[1] https://www.schneier.com/blog/archives/2016/10/security_econ...
As for the IoT, the appliances could have their firmware in ROM instead of flash. Then, the malmare would not survive a reboot. Many customers are large scale enough (Microsoft, Google, governments, etc.) that they can demand it of vendors and vendors will deliver. Really, how often do you desire to update the firmware on your hard drive, your USB stick, etc.?
(Another way to do it is to have the write-enable line controlled by a physical switch or jumper.)
The only thing I can figure is everyone has forgotten what ROM is.
Also, jumpers are cheap. Just set the jumper to enable writes, and download the update.
Are you happy with it being unknowable which of your appliances are compromised or not? Would you pay $1 more for a disk drive with firmware in ROM? I would. If you were running a banking system, would you pay extra for code in ROM that cannot be compromised?
As for IoT devices, I can't imagine average Joe or Jane prying off their smart whatever, destroying the pretty plastic casing it's hidden in, and manually setting jumpers to flash firmware.
I don't think the current scheme is working very well. Even worse, there is no way to tell if your appliance has been compromised or not. There are a lot of companies that care whether their machines are infected or not.
I think in all honesty the regulation sounds best, requiring the router makers to disable features such as UPNP would prevent massive attacks like these.
But then the device would be 'hacked' again within minutes of it coming back online.
Why? The rest of your comment doesn't address that.
>See Bruce Schneier's recent article on the matter, where he makes the case that government regulation is the only feasible remedy at this point
He also doesn't explain why. He literally writes:
>>In fact, the software used to attack Krebs was simple and amateurish. What this attack demonstrates is that the economics of the IoT mean that it will remain insecure unless government steps in to fix the problem. This is a market failure that can't get fixed on its own.
Why does it demonstrate that? "Neither the seller nor the buyer care" can also be applied to traditional botnets using computers - if the botnet is clever enough to not make any noise. He "justifies" the "sellers don't care" part with:
>>they're now selling newer and better models, and the original buyers only cared about price and features.
... again the same for smartphones or laptops. And furthermore, what regulation he proposes to solve this? I mean, he presents the issue as being:
>>The security of our computers and phones also comes from the fact that we replace them regularly. We buy new laptops every few years. We get new phones even more frequently. This isn't true for all of the embedded IoT systems. They last for years, even decades. We might buy a new DVR every five or ten years. We replace our refrigerator every 25 years. We replace our thermostat approximately never.
Would he suggest that the companies selling these devices be obliged to push security patches? The same companies that he wrote could not even afford testing?:
>>Companies like Microsoft, Apple, and Google spend a lot of time testing their code before it's released, and quickly patch vulnerabilities when they're discovered. Those companies can support such teams because those companies make a huge amount of money, either directly or indirectly, from their software -- and, in part, compete on its security. This isn't true of embedded systems like digital video recorders or home routers.
...
I have read "government is the only solution" at least 6 times in that short article, without exaggeration. Looks a lot like fearmongering and propaganda to me.
Not really. When a desktop PC is compromised sometimes it is noticed because eventually malware comes along that bricks the system, installs ransomware, or otherwise discloses itself.
But the sad truth is that a lot of compromised desktop PCs aren't noticed which is why there are millions used in botnets right now. Microsoft decided to take it seriously and pushed automatic updates, adopted secure coding standards, etc. It took a decade to make a good dent in the problem and it still isn't completely resolved.
>>they're now selling newer and better models, and the original buyers only cared about price and features. >... again the same for smartphones or laptops. And furthermore, what regulation he proposes to solve this? I mean, he presents the issue as being: >>The security of our computers and phones also comes from the fact that we replace them regularly. We buy new laptops every few years. We get new phones even more frequently. This isn't true for all of the embedded IoT systems. They last for years, even decades. We might buy a new DVR every five or ten years. We replace our refrigerator every 25 years. We replace our thermostat approximately never. >Would he suggest that the companies selling these devices be obliged to push security patches? The same companies that he wrote could not even afford testing?:
If all manufacturers have to meet certain standards then no one can undercut anyone else by skipping out. You can save some money by not including various safety features but then you can't sell those devices to over half the world so what's the point? If it is all voluntary you have busy consumers who don't have the requisite background knowledge (nor the time) to learn the details of software security, nor audit the code for secure coding standards and that's assuming the code was even available. If you want to make the claim that every human being should be an expert at evaluating such claims that's a massive inefficiency.
Really that's the primary problem with libertarian ideas in general: it introduces massive inefficiencies in the market. Not everyone can be an expert in chemistry, software, mechanical design, food safety, et al simultaneously. (Not to mention needing perfect and instantaneous information). Claiming that lawsuits will fix everything after-the-fact ignores the real human cost, the lost potential of the ruined lives, and doesn't work unless you completely eliminate the corporate veil and make everyone personally liable. Otherwise every company will just be incorporated with various shells to ensure you can't actually recover anything. That goes on today - get sued? File bankruptcy and re-incorporate under a new name. You don't even need to move the factory because the seller was just leasing it from your shell incorporated in the Bahamas.
>>Companies like Microsoft, Apple, and Google spend a lot of time testing their code before it's released, and quickly patch vulnerabilities when they're discovered. Those companies can support such teams because those companies make a huge amount of money, either directly or indirectly, from their software -- and, in part, compete on its security. This isn't true of embedded systems like digital video recorders or home routers. ... >I have read "government is the only solution" at least 6 times in that short article, without exaggeration. Looks a lot like fearmongering and...
lol
IoT companies have been pushing crappy proprietary solutions for a while now; mobile phones and computers are exempt from this primarily because they run a few major OSes, which can be relatively easily patched. IoT devices are a shithole of poorly written code that doesn't even work properly, let alone securely.
Not that big companies don't try to sell crap - but they can't so easily evade consequences, so they at least have to think about self-preservation.
As Wikipedia puts it:
> The evil bit has become a synonym for all attempts to seek simple technical solutions for difficult human social problems which require the willing participation of malicious actors
https://en.wikipedia.org/wiki/Evil_bit
But in a block chain, a Sybil attack—imagine voter election fraud, where numerous fake IDs are made to vote for a candidate—is blocked by making the cost of generating hash values (government ID is pretty hard to counterfeit) extremely high.
Can the same methodology be applied to blocked DDoS attacks?
Like using Dyn to find a domain is similar to making a phone booth call to yellow book call center, why not just raise the price of calling from 25c to $1.00?
In the first place, the requests are being made by owned IoT devices. So making the requests more expensive (in time, bandwidth, or actual money) would just hurt the owners of those devices. The people operating the botnet wouldn't care, they'd just get a larger botnet.
Is there a way to enforce a limit on requests from IoT, or a way to increase costs of running a botnet but not have that cost transfer to the owners of IoT?
Something I've wondered is how much of an impact large botnets have on other systems they rely on. If you had a million compromised CPUs in a small geographic area suddenly jump from a low power state into doing a massive amount of work, could it cause localized brown outs? Some napkin math says probably not, but it's not something I've heard considered much before.
Even very lame passwords might be expected to reduce the effectiveness of this attack approach by an order of magnitude or two.
But the problem remains that these devices, more often than not just don't get updated. So in a year or two, there will probably be a handful of exploitable issues that won't ever get patched...
Hopefully not many, though I could never understand why unattended-updates package is not installed and enabledby default.
I've never seen that happen in native GNU/Linux applications, though. I guess it might be related to the XUL stuff in Firefox.
But given that Firefox (and Chrome/ium) are memory hogs that eventually get OOM killed, you need to restart them periodically anyway, so it might not make a big difference if the OS does auto-updates in the background.
[0] http://www.telegraph.co.uk/technology/2016/08/25/windows-10-...
But it wouldn't solve the whole problem, just make it a tiny bit more difficult for the attackers. Definitely worth doing, but we will need to address the culture of 'ship and forget' also or we will just have the same problem but with software exploits.
Most better routers nowadays can already isolate devices from the internet/whitelist specific domains (family filter functions) and offer VPNs from the outside. From a technology perspective that's most of the pieces you need. Make a better UPnP implementation with user confirmation, make it easier to configure and everybody can get their devices nicely isolated-but-accessible.
That seems like the best solution. Now we just need to enforce that as a standard with internet connected devices.
Now if only we could get a standard router manufactures to stick to :-P
Sadly I can see IoT having to go though the same slow learning curve.
Just imagine how many scammers would pop up to 'secure' things for people locked offline.
Hacking a webcam would no longer lead to the ability to ddos websites, it would be a direct and total shutdown of the domestic internet supply.
Really this is about incentives: manufactures and users have little incentive to worry about security if the losers are third-party the targets of a DDOS.
Things will only improve when there is liability: either the manufacturers or the users get fined when their stuff becomes an attack vector. Fining users will probably never be politically possible, but I suspect it would be the better option.
That is fining manufacturers would lead to strict and expensive regulations saying just how things must be done, but which are laughed at by baddies developing new exploits that work within these particular rules.
On the other hand, if the rule is "if your box does bad stuff, then you must pay" -- then consumers will look to security as part of the good reputation of the vendor.
That extra layer of indirection is what I look for to avoid "last war" type scenarios.
Regardless of any good intentions, it'd be a pretty grey area to deploy a tool that cleaned and then protected against future attacks on a device. The exploit (well, at least the one currently drawing attention) is targeted at devices running BusyBox that are compromised over telnet with a embarrassingly small dictionary attack. If you could clean any existing instances, the protection for future exploits is simple - set a randomly generated password.
The downside is doing this (without approval) to remote devices would likely break device functionality. The majority of users/owners of these devices are likely completely unaware they have been compromised today as it has little to no effect on anything they can see. Sure, a robing hood malware could clean up this mess pretty effectively, but the little side effect of stopping devices from, you know, working is probably undesirable.
It's draconian, but it would sure make users sit up and take notice of which manufacturers are slacking on security.
Also, a lot of them expose web interfaces that Google has already found...
There is already https://twitter.com/internetofshit of course. But that is more whimsy and fun "Oh look my toaster is tweeting, that's silly".
This attack is more like "Oh-oh, my toaster killed Twitter".
So even though as a thing IoT is here to stay, I think the phrase itself will sound negative and will be avoided.
My thinking is that the only way to stop this is to get users to lock down their insecure devices.
A constant banner would annoy most people to action. Especially if it was really easy to issue a fix. Download and run. Script determines the actual device causing the harm and patches it. Possibly asking for a new password from user. I guess this assumes that a script could be written to issue a patch for the majority of cases. If not, then how is this device connected? Script could detect router and apply fix at that point.
So we need:
1. IP addresses for the sources of the attack load.
2. A way to distribute patches for those IP addresses.
I propose:
1. Those suffering the attacks can provide this.
2. Notify users via websites coordinating on displaying an alert with tools to patch.
Alternative solution to #2:
Can some part of the ISPs that connect these IPs be patched to detect and block malicious attacks?
Back in the day, the Internet was far simpler, with far less bullshit. If your IP was spewing crap, you would hear about it. And you either resolved the problem, or got null-routed.
The Internet is far^N larger and more complex, for sure. But accountability still has a place, I think.
I wonder if Dyn will at least release a heat map of source IPs.
In the US I believe cable modems are static DHCP leases tied to the mac address of the cable modem.
Of course this would be an arms race of sorts if the next version of the malware simply added port knocking sequences to the compromised routers in front of these vulnerable IoT devices.
I guess that wouldn't make your customers super happy, but it would probably be effective.
If that would somehow happen, you could use this miracle to solve most of the problems on this planet. This is literally the biggest issue of humanity - it's extremely hard to get people to coordinate. Doing so usually requires forcing various incentives to align.
In this case, you'd most likely need a direct government involvement to make all ISPs behave.
That's exactly what government regulation addresses. A truly global approach would be nigh-on impossible, but it should be possible to get all ISPs in the states to come on board, for example.
The monopolistic structure of the economy is a problem the US suffers much more than other countries (by definition at the cost of the consumer).
(Although the tendency to reinforce monopolies naturally applies to all countries.)
I'd be surprised if they aren't working on it right now.
[0]: https://idea.popcount.org/2016-09-20-strange-loop---ip-spoof...
">THEN google, Facebook, Twitter, etc (major web properties) use this list to notify any of its users that one or more of their devices have been compromised and point them to a how to guide for securing it."
I am not sure I want FB and Google policing the internet. They have enough power as it is. This would set a bad precedent.
It's not about the size of the infrastructure, it's about the size of the political statement. Targeting larger infrastructure first appears to have been a move to demonstrate the power of their botnet.
EDIT: this is the only realistic way of securing the devices.
A pity we can't have digital locks that 'break' when used, leaving some fingerprint from the perpretrator.
> Then privision a new box and try to ssh to it with port 22 blocked.
Shouldn't be a problem if you've got your provisioning stuff set to configure ssh on an alternate port.
> I could of course run fail2ban or a firewall rule in order to block at 3 failed attempts
A good idea, but still an additional step beyond simply using a different port.
> and still run on port 22
And you'll still end up on the receiving end of more malicious traffic than if you'd not used port 22.
> Or use port knocking.
Not necessarily a bad idea, but still far more complex than simply using a non-default port. Same reason that many ISPs (including Amazon) place more restrictions on port 25 than other ports commonly used for SMTP traffic.
- If you don't have a very well defined need for a short DNS time to live, set your time to live to a large value, perhaps a day. Then, as long as someone can get at least one DNS request through, they can reach your site all day. (Ycombinator.com, why do you have a TTL of 12 seconds?.)
- Get multiple DNS services now. Not just two. Get four or five, some of which are not widely used.
Those two things will probably get you through future DNS attacks.
Is this as simple as setting up the same records on multiple providers and updating your nameservers to point to the different providers? Or is there more involved?
Are there any providers which will replicate records from your 'master' provider, or is this going to be manual?
IIRC, bind has multiple built-in methods of keeping zone files synchronised between boxes.
You just need a primary DNS vendor which supports AXFR, such as DNS Park (https://www.dnspark.com/).
But I guess this excludes many of the popular DNS services that are used today.
Why is this not a default feature by DNS providers themselves?
I always assumed they act as slave copies for each other to avoid this kind of situations
The only problem is this seems be a tough one to get right config-wise and I just haven't had time to figure it out.
Anybody know of any good tutorials for this specific use case? I've googled it a few times and they never seem too easy.
While it's not exactly what you're looking for, you could use dnsmasq, you could use the "--min-cache-ttl" option to increase the TTL: but only up to an hour. You can patch the source to increase this time to suit your needs.
To exactly match your use-case dsnmasq would need to have a regular cache that respects upstream TTLs and a last-resort cache that doesn't.
You either meant to say 60 seconds, or forgot to factor your local caching DNS resolver in. Here:
That said, 60 seconds is still ridiculously low.No they don't. CDNs with the exception of AWS Cloudfront don't run on a cloud provider. Things aren't quickly spun up and down with any kind of velocity. CDNs manage all of their own infrastructure, POPs, hardware etc. There is usually a maximal hardware lifecycle since the CDN business is so CapEx heavy.
Cache-Control TTLs and DNS SOA record TTLs have nothing to do with each other. They are separate layer 7 concerns. The former is used by your browser and the latter by your system's resolver library.
I have never heard of anybody using SOA TTLs records to purge a cache. How would that even work?
CDNs generally give you decent tools to purge your objects.
Source: I worked for a CDN.
- Put up a caching DNS server on your local network to keep your regular sites functioning.
If you set a long DNS TTL, then you will not be able to move your IP addresses quick enough. In the DDoS world, you may need to nullroute an HTTP server IP. If you do that, you assume you can point the DNS to another, not-nulled IP. For this reason, keeping the DNS TTL to low values will make you more resilient.
Multiple DNS providers is probably a good advice, although hard to do in practice. The api's of different vendors don't always match. (cname flattening? geo-routing?). Also a real big attack will just likely take down all your providers. Using multiple providers may well _decrease_ the stability of the internet at large.
DDoS against DNS is not trivial problem. As sad as it may be, I'd say we must just assume DNS infrastructure in the world works. If it stops, then, many users will be affected.
Also hilarious:
>Anonymous didn't respond to a request for comment via Twitter