HN Explorer

Ask HN: What are some examples of “decentralized discoverability?”

35 points by mijustin ↗ HN
While the web itself is built on a series of decentralized protocols (TCP/IP, HTTP, DNS), discoverability is still centralized on platforms like Google Search, Twitter, YouTube, etc...

Even web3 stuff relies on centralized indexes/marketplaces like OpenSea.

The idea of mirroring an index (Pirate Bay), or having a distributed database (DNS), is fascinating to me.

These are the examples I can think of:

DNS UseNet FidoNet PirateBay Mastadon

Any others?

What could "decentralized discoverability" look like for content?

19 comments

[ 3.5 ms ] story [ 49.9 ms ] thread
[deleted]
(comment deleted)
dceddia
I've heard about CommonCrawl [0] a few times here and it seems interesting to think of building a search engine around it. Or, maybe, a boilerplate search engine that people could build upon. Maybe someone will even reply with an example :) I see they have a big list of projects [1] and honestly this might already be in there. It's fun to imagine a whole bunch of specialized search engines, like one that's great at finding product reviews, one for software development, one for recipes, and so on.

I think the powerful aspect of a distributed/mirrored index is that the data is public, so anyone can build a UI or a product around it to offer their unique spin.

From a more cynical angle, I also think it's likely that people still tend to gather around a few "popular" ones of anything like this. The web3 stuff seemed to do this. It happened with torrent trackers too. Eventually there's an option that is "base + X" where X is enticing enough to attract a crowd, and then once that crowd is big enough, that popular thing can pivot into a centralized thing, and the cycle repeats...

0: https://commoncrawl.org/

1: https://commoncrawl.org/the-data/examples/

mijustin
The difference with torrent trackers is their index is public and can be mirrored.

Torrent sites like 1337x were born out of previously popular indexes (Kickass Torrents, h33t). [0]

It's not dissimilar to when a GitHub fork becomes more popular than the original project. [1]

To me, the biggest disappointment with web3 is they never cracked "distributed discoverability." It still relied on the old model, of proprietary indexes/marketplaces.

I'm way more inspired by the P2P efforts, especially Gnutella's initial attempts at distributed search. [2] Feels like this tech could still be iterated on and improved.

0: https://en.wikipedia.org/wiki/1337x

1: http://gitpop2.herokuapp.com/tobi/delayed_job

2: http://rfc-gnutella.sourceforge.net/developer/share/intro.ht...

robcohen
OpenSea was popularized because the NFT minting functionality cost nothing for the artist. So all costs for the blockchain part was passed to the consumer. I believe this was the largest reason it became popular. Blockchain still doesn’t scale well, layer 2 is getting pretty mature though and with Eth Merge happening hopefully things will get significantly better in the next few years.
mijustin
The problem with web3 is that the projects still rely on centralized indexes/distribution (OpenSea, Rarible, etc) for discoverability.

"We have had decentralised ownership and distribution of our own content since the early 90s. I own my own domain, but that doesn’t get me traffic. The question is in the networks that sort, filter and organise that, and those aren’t our tokens that we can move." – Benedict Evans [0]

web3 can't deliver on its promise of "freedom from the big platforms" as long as we're beholden to big platforms (Google, OpenSea, Twitter, YouTube, etc) for distribution/discoverability.

The web3 world still hasn't figured out how to decentralize discoverability.

0: https://twitter.com/benedictevans/status/1476285185557774337

ratorx
BitTorrent had a protocol extension for DHTs, as a replacement for the trackers that are required by default: https://www.bittorrent.org/beps/bep_0005.html

So you still need a central web server to serve the .torrent file, but peers are found without centralisation. But I’m sure with effort that could have been generalised further.

There’s also IPFS (https://ipfs.tech), which uses content addressability to try to be even more decentralised than the normal web (without any cryptocurrency shenanigans, at least last I checked).

I don’t think either took off massively though.

anacrolix
Most BitTorrent clients support the ability to transfer the torrent info over the wire too: central trackers are not necessary at all.
anacrolix
I think IPFS development is funded significantly by Filecoin, a cryptocurrency.
h0p3
You can use links, webrings, and word of mouth. You can crawl (for non-web this may require quite specialized tooling), scrape, and compile a database for search (which can be done locally using whatever tools you prefer or even a full-blown webserver like gigablast or YaCy); and, to some degree, these practices themselves can be further decentralized and shared. Scaling is hard, especially if users expect immediate results or there are adversarial nodes to contend with.

Ideally, we could programmatically share feeds of curated and tagged information with each other that can be nested, mixed, or further shaped. Leveraging our trust in others (and their trust in others by proxy), which can be similar to some problems in distributed governance, is a p2p holy grail, imho.

mijustin
Iterating on the tech in P2P networks (outside of web3) still feels like it’s attainable.

Feels like a lot of the innovation stopped when Pirate Bay was repeatedly targeted, but there’s still lots possible in that space (outside of torrenting).

h0p3
I wouldn't say the innovation has stopped, but I will agree that the average person who uses the internet is far less likely to care about p2p tooling. It's also pretty hard to find people willing to construct exceptionally complex networks that cannot be monetized by their very nature.

A proper all-in-one library or binary for NAT Traversal (state of the art will only hit 95%) and next generation high-performance secure DHTs (secure in terms of privacy, anonymity, and resistance to attack) with multi-segmented transmission of data (bonding is critical for slow tunnels, bad peering, and maximizing the value of swarms), mutable content-addressable storage, baked-in obfuscated tunneling, and revokable + intelligent multi-device logical accounts (it's necessary to have our own devices and our friends proxy for us to solve some kinds of problems) is the missing foundation. Ideally, you'd be able to have lots of p2p applications sit on top of one network orchestration tool, otherwise you'll blow the average person's resources just trying to stay alive on multiple networks.

In applying the "decentralize until it hurts, then centralize until it works" mantra, I find we've still not attempted to decentralize as much as we should. Plenty of the experts I respect disagree with me there, so take my opinion with an extra grain of salt.

openfuture
In order to start doing this we need a standard to build on. The simplest thing should be the easiest thing to agree on and that has been the case so far in how math has moved forward (afaict) but sometimes local optimas get in the way of progress.
h0p3
`/nod`. I hear ya. RSS and mutable torrents seem like a fine starting place (if only to learn from), and I think we could argue ActivityPub's community has attempted to work on this problem. I worry that if we can't achieve the underlying distributed network (and effective DHTs [or the overlays on them] have a very long way to go, imho), we will lack the proper testing grounds to even know what the fitting standard(s) should be at some level. I hope I'm wrong about that.

I could see federated networks perfecting some aspects of self-sovereign, decentralizable identities or portable logical accounts, as well as fitting standards for sharing the data we find most salient among that federation. It's a start. I can't say something like Mastadon has succeeded here, but these amazing projects simply haven't received the funding they deserve. I could also imagine a centralized, for-profit institution creating something similar, but it would be a trade secret. I hope we find the answer before they do.

I'm not going to work on that problem until I have an underlying p2p network meant to actually replace at least the web (and, I am, of course, a huge fan of Sir pfrazee's astounding work in this sphere). I consider that an extremely tall task as it is, and, so far (and I say this with profound [[gratitude]] for those who have worked to solve the problem), nothing comes even close.

It's a pleasure to meet you, btw.

openfuture
Alright so you can go check out datalisp.is and tala saman then.. it's my attempt