FWIW I think the late Dan Kaminsky is the person who originally invented this two process userspace TCP scan technique 2001-2002ish in his tool Paketto Keiretsu.
It's a good one.
I sometimes wonder if it inspired some of the TCP-in-userspace stuff that is done in go (gvisor lib, I think it was).
I'm glad Rob is continuing the development of this idea and continues to scan the whole internet. Scanning the whole internet is cool. It's a shame it's de facto illegal these days (and will get most internet connections in the USA terminated near-instantly).
Decoupled sender/receiver scanners date back to the 1990s (we did one in CASL, our misbegotten packet programming language from 1997, the script for which I think shipped with Ballista --- and we stole the scanner design from someone else, probably from CORE SDI), as do userland TCP libraries (LWIP is a famous one, or at least a library that got used for that purpose, that predates all this stuff).
I know this came a few years later, but it really advanced the state of the art for widescale scanning at that time, particularly once it made its way into Kali.
For a long time, unicornscan was the preferred scanner for pentests (nmap is venerable and impressive but it has a lot of weird failure modes on real networks). I had no idea the author had died.
They mean a lot of ISPs and VPS providers will flag & terminate your account for launching port scans so it is difficult to find somewhere to run such a scan at any kind of speed.
Yeah, I appreciate his willingness to share "inconvenient truths" that may run counter to the running narrative of whatever tech/security story of the moment. Lots of interesting perspectives, even if maybe I didn't "want" to hear it ;)
Well, 5 minutes is about 2^8 seconds, you can get about 2^32 cycles per core per second nowadays, and you can have about 2^6 cores maximum, so with an incredibly, unspeakably generous "scan one IP in one cycle" a 64-core computer should be able to scan about 2^(8+32+6) = 2^46 addresses in a little under five minutes.
The IPv4 internet is merely 2^32, and people have been scanning that whole thing for years now. The /112 is referring to IPv6 addresses.
Plus the copy about not hammering other networks won't matter when you're trying to scan entire /64s that are behind one home router or something. That's gonna get noticed. You can't really scan IPv6 like you can scan IPv4, the math I gave is part of why. The code to do it is trivial, but the hardware just isn't there.
I'm really surprised shady figures aren't publishing lists of active IPv6 addresses to allow scanning.
For example, someone with access to a backbone internet router could easily log src and destination ipv6 addresses, and sell the complete list sorted and compressed. Malware authors could then use the list to portscan for badly firewalled stuff.
I'm pretty sure the IPv4 internet is constantly being scanned. I blocked ICMP echo request (ping) in my firewall, and saw it throws a warning of an attempted ping every other minute.
Yes, people often say this about IPv6 thinking they're being clever. It's not. 2^128 is big, relative to the universe itself.
It is an accident of history that computing spent so much time at binary orders of magnitude that technology could catch up and make the previous limits seem small. At some point, the exponential curve takes off and no amount of physical resources can catch up to it, because our universe taps out at O(n^3). 2^128 is pretty much on the other side of that takeoff point. You can almost catch up to it... at the cost of the resources of an entire galaxy... or more.
(Technically it's even sooner than O(n^3) if you also have to avoid black holes, but since the relevant comparison here is "exponential" the details hardly matter here.)
Fair point, I just did the math and even with a botnet of 13m (pulled from googling ‘largest botnet’) it’d still take e+22 years, give or take a few billion.
Still, network address obfuscation really shouldn’t be a security control.
Well you can understand why in the case of masscan, it would be a pretty boring claim to say that masscan is “scanning the entire internet in 100 years from a single machine!”
Aside from Shodan's neat trick of infiltrating the NTP pool, are there any papers/blogs on successfully exploiting known properties of IPv6 (network prefixes, MAC vendors etc)? However I think about it it seems too massive
I was mostly thinking about the sparse structure of upper bits (which are easily enumerable e.g. by downloading a RIPE database dump). As for the lower 64, MAC address OUIs contribute at most around 15 bits, and the distribution is likely strongly skewed towards only a handful of vendors, so actual randomness is probably lower still.
You still can, but that's for pentests _targetting_ AWS hosted infrastructure. They've always frowned upon using AWS to target _other_ services, however.
You can port scan your own infra/vpcs etc -- but YOU MUST tell them you are doing so and why you are doing it, else they will block it.
again, as with anything AWS (and other providers) have a good rapport with your rep, and SEs in AWS and you have a lot more freedom than you expect just from boilerplate ULA TOS stuff.
You may not scan anything other than your own infra. And you can get your external monitors whitelisted as well...
And if Amazon doesn't, are there any hosting providers that do allow port-scanning from their infrastructure? I'm assuming the big providers like AWS, Google and Azure all have these limits in place, but I wonder if you can do it from some of the smaller providers? Even if means going down to some fly-by-night outfit in Belize or something...
I'm always interested how much revenue peoples hobby projects earn.
The vast majority of projects seem to take someone months of work, get 50 github stars, and $25 worth of donations... Which is really sad - it basically means the vast majority of opensource authors have to have another job to pay the bills.
I just started to try and monetize. I'm a bit early, less than 50 stars. But my expectations are pretty much what you stated. Though my actual plan is to try and build a following like Jeff feeling and then sell training courses and ebooks.
Awesome tool. Rob added IPv6 support last year, which can be really useful if you know what you're doing (ex: want to scan a single subnet for given OUI without privacy extensions).
> This increases the rate to 100,000 packets/second, which will scan the entire Internet (minus excludes) in about 10 hours per port (or 655,360 hours if scanning all ports).
The Wireguard VPN protocol doesn’t reply to any packets that aren’t cryptographically authenticated, so that’s a good option if you want something “unscannable” on the internet.
For a regular, publicly accessible server, you do want people to be able to connect to it, so it must be scannable.
I don’t know offhand, but with HTTP/3’s udp based protocol, it ought to be possible to at least make scanning a bit trickier. By requiring a valid QUIC client hello packet, with a valid SNI header for that server, the scanner must know the name of the server it’s trying to talk to. I don’t have any experience with HTTP/3 yet so I am probably wrong.
Warning. Masscan is the best scanner I have ever seen.
We have also managed to take down the entire corporate network by using it with a too high rate limit. So tread lightly around massscan and its power. Our pentesters did the same 6 months later. The managed service provider is not able to solve the routing table loop which causes the firewall to DOS when a rapid masscan is triggered.
Yeah I managed to do the same, it filled up the firewall state table and nobody was able to initiate new connections, only existing ones continued to work.
And that's how I learned about stateful firewalls!
I once (2016) used it to scan port 22 on the whole ipv4 (had to experiment with rate limits to not trigger alarms and get complaints forwarded by my VPS provider; the clever ip+port randomization technique helps a lot with that). Then took the ~22m (iirc) IPs which responded and ran ssh-keyscan on them to extract and analyze some ~15m ssh banners and public keys (a bunch of them broken, through debianized PRNG etc.) I think most of the scanning + extraction was done overnight, via ~13 VPS rented hourly (whole thing cost < $10, and very few complaints). Fun times :) I should write it up some time, and do it again.
"mm" is a common abbreviation for million in the context of accounting. It refers back to the latin "mille mille" which means "thousand thousand" literally. "Mille" is still used for thousand in French and Italian. The abbreviation is reasonably common in english, though I also would not be surprised by anyone not having run into it before.
The Latin numeral M denotes thousands. Thus, MM is the same as writing "M multiplied by M," which is equal to "1,000 times 1,000", which equals 1,000,000 (one million).
So, we should refrain from using lowercase m for this.
Your quote left out the first part, which is italicized in the full quote below:
In finance and accounting, MM (or lowercase “mm”) denotes that the units of figures presented are in millions. The Latin numeral M denotes thousands. Thus, MM is the same as writing “M multiplied by M,” which is equal to “1,000 times 1,000”, which equals 1,000,000 (one million).
Why do you think it's in parentheses? Because it's not the main, correct way to write this. Lazy people didn't capitalize, and now you see it in the wild. Given the origin is Latin numerals, I see no argument here.
Lower-Case use of mm in accounting and finance is incredibly common. If you worked in those industries it’s understandable you’d think everyone followed that convention.
"A very nice and well-built tool. .... (very well-documented source too, a pleasure to read)" Could not agree more. This code is really nicely organized and commented. Such a pleasure to read and learn about how this tool works. I wish more codebases had comments like this! Great work and thank you @ErrataRob.
Question: how do you actually run it without getting banned?
I've got banned pretty hard by both my local home ISP and using Linode servers, when tried such scanners. Mass port scanning is easy to track, and it usually forbidden all ISP ToC.
I did a CTF a while back and it was used to find the real IP of a missconfigured .onion site. I didn't actually do it as I was worried about burning a VPS provider.
My ISP had a misconfiguration that let all the customers ping each other (Routers usually). The address space was too large to scan any other way except masscan, which worked very well (and fast).
Because it sends a lot of traffic in such a small time frame. Also, it has obvious nefarious purposes and the ISP may face legal consecuences in case someone lawyers up.
A quick plug for my friends over at GreyNoise... they have honeypots all over the internet and identify various scanners as well as their observed intentions.
Hey there I manager our Community at GreyNoise - those are the Organizations that the IP is originating from! Our "Actor" Classification gives better insight into /who/ is actually conducting the activity (i.e. https://www.greynoise.io/viz/ip/92.154.95.236)
You can look up IPs in GreyNoise to know if they are scanning the internet (if no results are returned, the answer is likely no or not at scale.)
>"A mutex on the fast path of a program severely limits scalability. Instead, Masscan uses "rings" to synchronize things, such as when the user-mode TCP stack in the receive thread needs to transmit a packet without interfering with the transmit thread."
Is "rings" here referring to PF_RING mentioned in the preceding paragraph or is it referring to a specific synchronization primitive?
Rather than have everyone who can run this (without getting banned) actually run it, couldn't someone just post the results to a file for us once per day or week? Seems inefficient to have millions of people run this when it results in (mostly) the same data.
I would have fun optimizing this. The performance discussion toward the end of the readme was pretty basic, no assembly language or multi threading optimizations, etc.
But I have told myself to stop spending time optimizing things for no reason..
Massacan is awesome. One of the usecases is to periodically scan your own servers to see if you have not accidentally opened some new ports in firewalls.
138 comments
[ 913 ms ] story [ 699 ms ] threadhttps://news.ycombinator.com/item?id=8803498
Some idiot is using your tool to mass scan our network - https://news.ycombinator.com/item?id=24728123 - Oct 2020 (182 comments)
MASSCAN: Mass IP port scanner - https://news.ycombinator.com/item?id=12260809 - Aug 2016 (33 comments)
Masscan: Scan the entire Internet in under 5 minutes - https://news.ycombinator.com/item?id=8803498 - Dec 2014 (29 comments)
Masscan: scan the entire Internet in under 6 minutes, 10 million packets/second - https://news.ycombinator.com/item?id=6391266 - Sept 2013 (30 comments)
Masscan: The entire internet in 3 minutes - https://news.ycombinator.com/item?id=6388222 - Sept 2013 (12 comments)
It's a good one.
I sometimes wonder if it inspired some of the TCP-in-userspace stuff that is done in go (gvisor lib, I think it was).
I'm glad Rob is continuing the development of this idea and continues to scan the whole internet. Scanning the whole internet is cool. It's a shame it's de facto illegal these days (and will get most internet connections in the USA terminated near-instantly).
[1] https://defcon.org/images/defcon-13/dc13-presentations/DC_13...
I know this came a few years later, but it really advanced the state of the art for widescale scanning at that time, particularly once it made its way into Kali.
Ehhh... I forgot he was the "the late Dan Kaminsky"
:-(
2021 has not been a great year. Too many, too young.
https://en.wikipedia.org/wiki/Dan_Kaminsky
I've not seen anything that says simply scanning for open ports is illegal, doing vulnerability scans may be though.
He often discusses current events in infosec if you're into that.
We seem to be a wee bit short of 2^126, yes.
Plus the copy about not hammering other networks won't matter when you're trying to scan entire /64s that are behind one home router or something. That's gonna get noticed. You can't really scan IPv6 like you can scan IPv4, the math I gave is part of why. The code to do it is trivial, but the hardware just isn't there.
For example, someone with access to a backbone internet router could easily log src and destination ipv6 addresses, and sell the complete list sorted and compressed. Malware authors could then use the list to portscan for badly firewalled stuff.
There’s more than enough stuff on IPv4, IPv6 isn’t worth the effort.
FWIW shodan was setting up their own public ntp servers to track down v6 users.
First time to be called shady, though ;)
It is an accident of history that computing spent so much time at binary orders of magnitude that technology could catch up and make the previous limits seem small. At some point, the exponential curve takes off and no amount of physical resources can catch up to it, because our universe taps out at O(n^3). 2^128 is pretty much on the other side of that takeoff point. You can almost catch up to it... at the cost of the resources of an entire galaxy... or more.
(Technically it's even sooner than O(n^3) if you also have to avoid black holes, but since the relevant comparison here is "exponential" the details hardly matter here.)
Still, network address obfuscation really shouldn’t be a security control.
Oh my, that is clever =)
All I could find was statements from them that they do not allow port scanning OF their infrastructure.
You can port scan your own infra/vpcs etc -- but YOU MUST tell them you are doing so and why you are doing it, else they will block it.
again, as with anything AWS (and other providers) have a good rapport with your rep, and SEs in AWS and you have a lot more freedom than you expect just from boilerplate ULA TOS stuff.
You may not scan anything other than your own infra. And you can get your external monitors whitelisted as well...
Just talk to your rep.
The vast majority of projects seem to take someone months of work, get 50 github stars, and $25 worth of donations... Which is really sad - it basically means the vast majority of opensource authors have to have another job to pay the bills.
So 655360 hours, not 5 minutes
For a regular, publicly accessible server, you do want people to be able to connect to it, so it must be scannable.
I don’t know offhand, but with HTTP/3’s udp based protocol, it ought to be possible to at least make scanning a bit trickier. By requiring a valid QUIC client hello packet, with a valid SNI header for that server, the scanner must know the name of the server it’s trying to talk to. I don’t have any experience with HTTP/3 yet so I am probably wrong.
We have also managed to take down the entire corporate network by using it with a too high rate limit. So tread lightly around massscan and its power. Our pentesters did the same 6 months later. The managed service provider is not able to solve the routing table loop which causes the firewall to DOS when a rapid masscan is triggered.
I once (2016) used it to scan port 22 on the whole ipv4 (had to experiment with rate limits to not trigger alarms and get complaints forwarded by my VPS provider; the clever ip+port randomization technique helps a lot with that). Then took the ~22m (iirc) IPs which responded and ran ssh-keyscan on them to extract and analyze some ~15m ssh banners and public keys (a bunch of them broken, through debianized PRNG etc.) I think most of the scanning + extraction was done overnight, via ~13 VPS rented hourly (whole thing cost < $10, and very few complaints). Fun times :) I should write it up some time, and do it again.
22 meter?
22 millies of something.
I do not follow?
“People don’t understand”, none of those were at all plausible even without the context of scanning the entire IPv4 address space.
So, we should refrain from using lowercase m for this.
Except that mm is already used by a number of industries and people to mean million.
What you did was quote from the first result of a google search without attribution and then pretend the quote means the opposite of what was written. See: https://corporatefinanceinstitute.com/resources/knowledge/ot...
Your quote left out the first part, which is italicized in the full quote below:
In finance and accounting, MM (or lowercase “mm”) denotes that the units of figures presented are in millions. The Latin numeral M denotes thousands. Thus, MM is the same as writing “M multiplied by M,” which is equal to “1,000 times 1,000”, which equals 1,000,000 (one million).
0: https://github.com/zmap/zmap
0: https://github.com/zmap/zdns
I've got banned pretty hard by both my local home ISP and using Linode servers, when tried such scanners. Mass port scanning is easy to track, and it usually forbidden all ISP ToC.
https://medium.com/@captn3m0/i-scanned-all-of-act-bangalore-...
Here is everyone running masscan against the internet: https://www.greynoise.io/viz/query/?gnql=tags%3A%22Masscan%2...
You can look up IPs in GreyNoise to know if they are scanning the internet (if no results are returned, the answer is likely no or not at scale.)
Happy to answer any q's :)
I went back to the page linked by the GP post. I clicked through all 10 items and the actor was "Unknown" on all of them.
I don't really expect you guys to be able to know who is renting AWS servers or something. I don't know how that would be possible, honestly.
It's a cool site, though.
>"A mutex on the fast path of a program severely limits scalability. Instead, Masscan uses "rings" to synchronize things, such as when the user-mode TCP stack in the receive thread needs to transmit a packet without interfering with the transmit thread."
Is "rings" here referring to PF_RING mentioned in the preceding paragraph or is it referring to a specific synchronization primitive?
But I have told myself to stop spending time optimizing things for no reason..
https://github.com/bobek/masscan_as_a_service