HN Explorer

Ask HN: I've Built a DHT Torrent Sniffer and Search Engine. Should I Release?

165 points by sylwester ↗ HN
Recently I was researching about DHTs and developed a DHT Sniffer in Go which connects to some known DHT Routers and sniffs all the annoucements. I've quickly added ZincSearch and it is now basically a search engine which can search for hashes, name or files contained in the torrents. It is able to index around 5-10k annoucements per second, so the index grows quite fast.

Now, I am thinking about releasing it as open-source for others to study, but not sure if I should, because it might be used for "evil".

104 comments

[ 3.7 ms ] story [ 210 ms ] thread
compressedgas
Consider that such already is available as open source in Go even.
navjack27
That would be very useful... Just release the code and building instructions.
2Gkashmiri
yes please. you are not responsible for any "Evil" users of the software might do. This is not even about enabling bad stuff, this is just natural progression of technology.
rrwo
Nuclear and biological weapons are also part of the "natural progression of technology" but it's widely agreed that they shouldn't be released to anybody and everybody, and the people distributing those technologies have a moral responsibility.

If you think your software would be of more use to "evil" than not, then don't release it widely.

loa_in_
Yeah, it is everyone's moral responsibility to consider effects of our actions as far as we're able to.
qualudeheart
Which evil usages are you concerned about? I think it would be very useful for the public.
seqastian
copyright holder bots
Ysx
DHT has existed for 17 years - the cat's out of the bag. The anti-piracy companies have built their own crawlers.
aliqot
They match swarm IP's and then notify ISP's who voluntarily hassle you on their behalf. An IP doesn't equal a person in USA.
netsharc
And outside of the US? In Germany there's an "efficient" business of law companies acting on behalf of the copyright holders, subpoenaing the ISP to get the owner of the IP/connection and sending them a cease-and-desist with a hefty fee.

There's the next part, law companies specializing in "Here's the letter you have to send to say you agree to pay them without admitting guilt, that'll be € please!".

TillE
I think the demand is usually around 1000 Euro, so it's way cheaper to just pay for a VPN service, configure qBittorrent to only use your VPN interface, and torrent whatever the hell you want.
0dayz
I would recommend writing down the worst and best case scenarios that could happen with your software, then determine if you notice either that through severity or quantity the software outweighs the positives, don't release it.
tmtvl
Just AGPL it, I hear it's an effective ward against Alphabet.
Eduard
Very good suggestion, considering the sniffer could be used by copyright claims lawyers.
MattPalmer1086
Can't see how AGPL would stop anyone using it. Doesn't it just prevent them from modifying the source code without sharing their changes. ?
webmobdev
AGPL fixes a "bug" in GPL that many tech companies have exploited to not release modified source code as the GPL requires. Simply put, GPL says that you have to release the complete source code (including any changes you have made) of a GPL licensed code only if you distribute it to other users. Many tech companies thus avoided GPL code. But with the growth of Software-as-a-Service, where an application only runs on the server and is accessed through a browser or an app, many of these same companies created web applications with GPL code that they customised. However, if a user demanded the source code of the GPL code, along with the modifications they made, the tech companies refused to provide it claiming they weren't "distributing" the application (as in giving you the whole application to run on your computer). And since they weren't doing that they claimed they had no legal obligation to release the complete source code.

AGPL fixes this - it recognizes SaaS web applications too as a "distribution" of GPL software. So if a source code is licensed under AGPL, anyone who uses it to create web applications and makes it available to the public is now also legally obliged to provide the complete source code if any user requests for it. (And ofcourse, as with GPL the user is free to use the source code as they want, freedom to improve it, and even start competing SaaS services).

That is why the AGPL is currently the best GNU license to ensure that your open source code always remains open source.

Shadowed_
How would someone know if SaaS uses some AGPL code?
kruuuder
In general, you don't, but that's a different question.
webmobdev
All AGPL / GPL licensed code requires that the AGPL / GPL license also be distributed with the application. Moreover, AGPL / GPL also requires public attribution (acknowledgement / credit) of those who have created and contributed to the source code.

Ofcourse, if somebody wants to be dishonest and unethical, they can violate these terms to hide the fact that they are using AGPL code in their SaaS application. There's not much you can do about it in such cases. But if anyone (like current or former employee) leaks this info to the public, they can face legal trouble and bad PR in the future as it would be akin to using pirated software.

remram
It only prevents them from modifying the code and not sharing their changes and making the service available to another company.

You are still free to make private changes to AGPL code and run it for yourself (or your company).

Eduard
I'm not considering "anyone" here, I'm considering copyright/license lawyers.

Would be kinda dumb for them to violate a license agreement (AGPL) while hunting for license violations.

remram
AGPL wouldn't stop copyright lawyers from using it or even making changes and not sharing. It would only stop another company from making changes and not sharing them and selling the service to lawyers.
Eduard
Of course copyright lawyers have to prove in court their means how they hunted some violation.

They would therefore have to disclose any modifications they made to an AGPL software, therefore giving insights how they hunt, therefore allow for better countermeasures.

Don't be naive about lawyers, they are aware what AGPL means.

remram
> They would therefore have to disclose any modifications they made to an AGPL software

That's just not what the license says.

brobinson
Let it rip. DHT has been around for so long now that whatever bad actors/evil use cases you're imagining have already happened. It sounds like a cool project, and I'd be interested to see it.
sshine
I accidentally read this as "Let it R.I.P."

I totally agree, unroll it!

While the file sharing is distributed, the centralised web-based indexing is still a game of whack-a-mole.

MuffinFlavored
how is what OP made not basically going to be the p1rat3 bay 2.0?
r3trohack3r
It already exists. BTDigg: https://en.m.wikipedia.org/wiki/BTDigg

And sibling comments show there is a community of folks doing this.

People also do it to Tor’s DHT too for discovering hidden services: https://donncha.is/2013/05/trawling-tor-hidden-services/

You can’t really stop open source tech like this from existing by choosing to not open source your implementation. You just end up staying out of the conversation yourself. The community will still build it if there is a desire for it.

haswell
One is a repository of code that you or I can run on our own machines.

The other is a publicly accessible website that got a kit of public attention because it could easily be used by anyone including novices to pirate content.

thomastjeffery
Moderation.

TPB (and other trackers) are essentially forums.

pdimitar
I imagine any bad actors who store IPs of torrent seeders have done so a long time ago already so your software will not do any harm that hasn't been done already.

Go for it and open-source it.

sascha_sl
I mean, it already exists. [1] Always fun to see what my neighbors behind the same NAT download.

[1]: https://iknowwhatyoudownload.com/

oefrha
Ah, haven’t checked that one for a while. Surprised to find a tech-minded neighbor who has downloaded kali-linux-2022.3-live-everything-amd64.iso. And no pr0n this time round.
IYasha
You'd better have a good firewall since you have such a tech-minded neighbor. )
insane_dreamer
I thought streaming had spelled the demise of torrenting, so was v surprised to check out my local hood (quiet suburban, skewing older few ppl <40yrs) and see how it's very much alive and well. I guess I was naive. (Not condemning anyone, as I did my share of peering back in the day, but streaming services are convenient and honestly don't cost that much all things considered. Torrents (tho I prefer IRC) good for finding obscure stuff that's unavailable anywhere else, but that's not what my neighbors are downloading; and it's not like any of them can't afford to pay. Interesting dynamic.
Klaster_1
It gives quite fascinating results for me. Surprisingly, among dozens of torrents only a couple are porn. Most of my neighbors using BitTorrent are Russians and Ukrainians (I'm in Cyprus), judging by files names and seeder distribution, a couple of English-speaking people (probably brits) and a single person seeding content in Greek. At least half of Russian-speaking neighbors seed IT, engineering and English learning materials. And I seed more than everyone together on the same subnet Wish I could message them.
imhoguy
Checked it for my Transmission host static IP which doesn't download anything for a while and it has some completely random stuff I never downloaded or searched for. Even shows items from today which for sure I haven't requested. Could be some DHT spoofing or my Transmission is relaying some DHT requests?
[dead] synctext
dhaavi
Tribler is an interesting project, but not directly comparable to what the OP is trying to achieve.
synctext
indeed, bit different. lots of people worked in this field and made these tools. Around 2014 the state of the art was 20k DHT responses per second, see '100 million DHT replies'. https://doi.org/10.1109/P2P.2014.6934318
dhaavi
Thanks for the pointer!

I just read some interesting proposals in the GH issues and recognized your username. This is some nice piece of work - will definitely return to read more about the underlying concepts.

We are working in a similar area with similar problems: Building a new Tor/VPN-like privacy network. See https://safing.io/spn/

simjnd
Taking this chance to say thank you for Safing Portmaster it is awesome!
pharmakom
Who is funding it and why?
someotherperson
If you follow through to the site linked in the Github:

> Tribler is a research project of Delft University of Technology

> Work on Tribler has been supported by multiple Internet research European grants. In total we received 3,538,609 Euro in funding for our open source self-organising systems research. Roughly 10 to 15 scientists and engineers work on it full-time. Our ambition is to make darknet technology, security and privacy the default for all Internet users.

> Vision & Mission ... "Push the boundaries of self-organising systems, robust reputation systems and craft collaborative systems with millions of active participants under continuous attack from spammers and other adversarial entities."

IYasha
The "robust reputation systems" made me twitch a bit...
synctext
See the latest 2022 algorithm called MeritRank [1]. Improves on the prior work developed together with Harvard. We deployed roughly 5 generation in the past 17 years. [1] https://arxiv.org/abs/2207.09950
peterkelly
OP said they already have a working system. How well it works remains to be seen, until it's publicly available.

On what basis do you assume he/she has not succeeded without even having seen their work?

synctext
We tried too. Several people have made DHT sniffers, including our lab. You see lots of spam and get stuck. Filtering out the noise is weirdly hard.
simjnd
So you're assuming they cannot possibly have succeeded simply because you failed?
synctext
these things need spam measures to work. That was not listed as a feature.
have_faith
> We are building a micro-economy without banks, without advertisers, and without any government

But with government funding?

BLKNSLVR
I'll add to the chorus of people saying "yes, release it".

If you're worried about blowback as a result of "evil" uses / users, is there a way to release it (somewhat) anonymously, so it's difficult to be traced back to you?

icpmoles
Is it basically btdig.com ?
storrgie
An advantage here would be people self hosting their own based on this project.
thinkmcfly
I think you should. From my understanding, use of DHT is already dead in the eyes of most torrenters
gwnywg
I was planning to start learning GO, I'd be definitely interested to learn from your project :)
boramalper
I had been working on this successfully for a couple years in the past before I got tired of it and moved on. I still think it's a magnificent idea, to be able to host your own torrent site and to decentralise the last centralised bit of BitTorrent.

https://github.com/boramalper/magnetico

hardwaresofton
Thanks for writing that note on the repository -- it was thoughtful and well explained -- reading through your small manifesto that was linked and that's fun too.
kevinwang
I think the manifesto was Aaron Swartz's
hardwaresofton
oh you're right I didn't even notice -- aaronsw.com... First time I've read it.
rakoo
I've built a small cli utility to search the database: https://sr.ht/~rakoo/magneticos/. It also asks the DHT and known trackers for the liveness of the swarms, so I can sort by seeders. It's simpler and more useful than running a full webserver with ports and all when I only need an ssh access. It has served me well and will be more than enough for a long time
thomastjeffery
The main thing that seems to be missing, in my experience, is moderation.

If someone can manage to make a decentralized forum-like moderation list, then we wouldn't have any use for centralized trackers anymore.

I think it could probably be done by using public GPG keys as identities, then keeping changes in a torrent-distributed git repo where user-generated data is saved in GPG signed git commits.

Then the only centralization left would be what branch/torrent people introduce to new users.

mdaniel
> I think it could probably be done by using public GPG keys as identities, then keeping changes in a torrent-distributed git repo where user-generated data is saved in GPG signed git commits.

Congratulations, you won some HN bingo squares by introducing both PKI and blockchain to the discussion :-D

But, in all seriousness: moderation against what?

soulofmischief
Yes, moderation is as simple as not downloading something you object to. As for identifying false metadata, that falls outside of moderation.
thomastjeffery
Not against. For.

For curation. For categorization and deduplication. And for trust.

You can only express so much with titles and seed/leech counts. Trackers that were known to be the best - like What.CD and waffles.fm - were private and heavily moderated. You had to be invited, usually by passing a quick interview to prove you knew how to avoid common data management pitfalls like transcoding from a lossy codec. That made those trackers into carefully curated collections of high quality content.

Even public trackers benefit from some moderation. Most people only want to be browsing porn intentionally, so having a category for it let's you browse for movies or whatever. It's all about sorting the noise into signal.

It's also helpful to have some commentary on a torrent. Maybe the subtitles are forced. Maybe the software has a virus or includes adware, but ended up getting a critical mass of seeds/leeches anyway. Maybe there is some similar content worth talking about.

Have you ever used a public decentralized platform that wasn't full of trolls? I haven't.

EVa5I7bHFq9mnYK
>> then we wouldn't have any use for centralized trackers anymore.

So we finally invented emule?

Centralized servers that provide selection, moderation, quality control, information, imdb score etc, is exactly why bittorrent protocol won against all the decentralised protocols.

thomastjeffery
I'm not familiar with emule, but as far as I can tell, its credit system is only based on seeding or uploading new content.

That can be useful, but it isn't the moderation I was thinking of. See my reply to the sibling comment.

EVa5I7bHFq9mnYK
I dont remember of any credit system in emule, people upload and share just out of generosity. I guess it also contributed to bittorrent success - uploaders probably get some share of tracker site ad revenue.
majestic5762
I wrote a similar solution 8 years ago. I repurposed the system to identify IPs owned by the government, and notified them if a malicious copy of Windows (but not limited to) was seeded by them. Meaning there was a chance that an unknown actor had a backdoor in my government's network. If you wanna discuss, I'm happy to talk and even contribute towards a commercial solution majestic.hn@fastmail.com. I didn't pursue this opportunity for money at that time, but I had my fair share of "shoutz". Won't be bad to pick this up again
[dead] zelienople
computerfriend
I don't know why you think the OP wants to be regarded as a professional. But you'd do well to adopt a little professionalism in your communication too.
jrm4
Please do. This old-ish timer is kind of blown away by the idea that you shouldn't.

Nothing evil about being a modern archivist/librarian, despite what big companies would tell you.

hardwaresofton
Would you mind explainng why you chose ZincSearch? Curious on why you picked it over some of the other non-ElasticSearch/OpenSearch alternatives (Meilisearch, Typesense)
keroro
torrent-paradise [0] is a go project which seems to do the same as your project and has existed since 2019. It’s since gone down but remains up on IPFS [1] but its index hasn’t updated since January.

[0] https://github.com/urbanguacamole/torrent-paradise

[1] https://cloudflare-ipfs.com/ipfs/QmQjsKamNFZRvCMXDvZXQmRYjsm...

hombre_fatal
This is a good example of our hubris as developers. We like to think our project will have some impact on the world when in reality you’re extremely lucky if anybody notices much less cares. ;)