The Internet may need P2P file sharing protocol, but that protocol never be IPFS. BitTorrent is the closest practical protocol currently used. IPFS is just a joke.
IPFS is just ... fine. It more or less works, but it's not very good at what it does, and the only advantage it has over BitTorrent v2 is marketing and web gateways.
I think the biggest complaint is performance, which is reported by some users to be amazingly poor (to the point that such a user would call the whole thing a joke on that account). YMMV.
That's because everyone has forgotten the lessons from BitTorrent and the internet.
If you want something to be fast then your content needs to be hosted on servers with fast connectivity to other servers. And that's not possible when at the same time you're trying to avoid legal jurisdictions that will censor e.g. EU, US, CN etc.
If you want something to be fast you need it stored locally. Storage used to be big and expensive, thus the cloud. And network used to be slow and expensive too, so the uplink direction wasn’t very effective unless there was a company behind it.
Which is to say - the main technical problems that hampered BitTorrent probably aren’t there anymore.
It is bad for universally legal uses (sharing, mirroring) because centralised alternatives are always faster and cheaper (benefiting from economies of scale).
It is also probably (I'm not quite sure yet) worse for the illegal uses because it is slower and more compute heavy than torrents, and there is way worse discoverability and moderation, which is important in the communities sharing illegal files.
Basically any true decentralisation is an attempt to eliminate trust from the system. And there are two problems with this - 1) maybe 0.1% of all attempts are actually decentralised (other 99.9% are either self delusional or lying), and 2) trust is actually a desirable thing, especially in the important things like money or file storage.
With IPFS, you don’t have to have any data on your computer you don’t want. No one uploads the data to unsuspecting strangers, they announce the content (it uses content addressing, hashes). Then other people choose to download it.
You can’t possibly hand approve everything so I’m sure the vast majority of node runners auto approve stuff and perhaps share some (non decentralized) black lists.
>You can’t possibly hand approve everything so I’m sure the vast majority of node runners auto approve stuff and perhaps share some (non decentralized) black lists.
This obviously hasn't been the case for bittorrent, so I don't know why you would assume that would be the case for IPFS. Even ignoring the legal risk of blindly re-hosting whatever illegal bootleg-ISIS-child-porn-malware.exe on the net, the use of a blacklist isn't sybil proof so you just get bogged down with spam by the first attacker and run out of disk space.
Popular files gain more hosts seeding. Rather than a page getting hard to access when its getting slammed with viewers, it should in theory become even faster.
You don't add them manually, its just when you open up a link in your browser or video player, it gets stored for some amount of time and seeded before perhaps being pruned or kept forever depending on the users preference. You don't have any issues with unknowingly hosting illegal files this way.
At least read the article.
"But is there a dark side to decentralization? We often hear concerns about how peer-to-peer networks may be used by bad actors to support illegal activity. These concerns are important but sometimes overstated.
One area where IPFS improves on HTTP is in allowing comprehensive auditing of stored data. For example, thanks to its content-addressing functionality and, in particular, to the use of unique and permanent content identifiers, IPFS makes it easier to determine whether certain content is present on the network, and which nodes are storing it. Moreover, IPFS makes it trivial for users to decide what content they distribute and what content they stop distributing (by merely deleting it from their machines)."
I don't understand this position. The main use case is for serving machine learning models and similar types of data.
A big use case is for researchers who don't want to constantly spend money to try to maintain a link to something they did at an institution they are no longer a part of. Many times they don't even have a mechanism to do so - so the link rot is guaranteed because of centralized systems.
People want HA, p2p will never work. As soon as you introduce nodes or machines that serve to increase availability you might as well drop the p2p and make the system more efficient.
I suppose it really depends on what your definition of "reliable enough" is. I've not used apple music, but as a consumer I don't remember the last time I was impacted in any significant way by a YouTube issue. If we have a couple minutes downtime a year from those services, like we seem to right now, I don't really see it as a problem.
There are services where I don't want them going down ever of course. But those are more critical infrastructure than something like Apple Music or YouTube.
i love p2p and torrents but ipfs really looks like a solution looking for a problem. i still havent seen a user friendly implementation of it usable by the masses. (yes if you know some feel free to comment it, of course id love to check it out)
It doesn't, thats just a mirror and even then everyone just uses the cloudflare or ipfs.io gateway instead of actually using IPFS.
Dont get me wrong I like IPFS and would love to see increased usage but if a libgen mirror is the biggest example then we need to find more applications for it.
You will start running into exactly the same problem as with normal hosting. You need someone to “pin” the music for you, or you pin it yourself but with public IP and will get sued.
Unlike with torrents, most people will be using the “gateways” - most people are not “seeding” - partly because IPFS itself is resource hungry and hard to use, unlike Torrent which is easy to use.
In reality IPFS world is very centralized, you just have this resource hungry pretend-decentralisation on top.
I don’t know why is it like that when BitTorrent is much older and much more actually decentralised.
Avoiding the legal problems seems doable, Sci-Hub and LibGen are managing. But the gateways thing sounds... unfortunate.
The way BitTorrent fixed this was the ratio - if you weren't seeding enough, you'd eventually get kicked out from the tracker. Does IPFS have a similar mechanism, or something else but with the same purpose?
In recent years there has been a large growth of alternative IPFS implementations. Partly because of different needs, partly because of unhappiness with the performance and reliability of the go-ipfs implementation.
At my previous company actyx.com we wrote our own ipfs implementation in rust called ipfs-embed ( https://github.com/ipfs-rust/ipfs-embed ) that serves as the storage for our distributed event sourcing system. It is used in production and is reliable.
The user of an Actyx deployment will not notice in any way that they are using ipfs in the background.
There are also some other interesting developments involving ipfs and rust. See a list of implementations here:
I will talk about it at IPFS camp 2022. I hope it will be recorded.
Here is my talk about actyx from IPFS camp 2019: https://www.youtube.com/watch?v=Qzu0xtCT-R0 . It does not go into much technical details, and it does not describe the current architecture but a previous iteration though.
ipfs-embed is using rust-libp2p and official protocols like gossipsub. So for that it is fully compatible with the ipfs network. At least as compatible as you can get when the specs are sometimes vague.
Bitswap has a compat mode, but when two ipfs-embed versions talk to each other they use a different protocol.
Note that ipfs-embed is made for a different use case than the global ipfs. It is typically used by private swarms where there are a smaller number of nodes. So e.g. we don't use the DHT at all.
We also found that go-ipfs compat is not that useful for our use case, so the compat layer was mainly done as a service to the community and might not work with the latest go-ipfs / kubo.
Makes sense, thanks! I'm not sure what not using the DHT entails, does that mean you can't discover content? That can't be it, so how does that happen? Is it just gossip?
Basically the nodes of the swarm (around 100 at most) gossip events to each other. That is also how they learn about the presence of the other nodes. They try to stay connected as well as the network topology allows.
Then when you want some bulk data via bitswap they just ask their current peers. Either the emitter of the data is a current peer, then it obviously has it. Or eventually one of your peers will replicate the data, and then you will get it with some delay.
The latter happens frequently if a part of the swarm is behind a NAT. You have some delay, but eventually everybody gets all events.
It is usable on WinWorld. Here's a link to a download so you can test it yourself if you like Windows 3.1 version 3.01.068 beta from 2-3-1992 on 5.25-1.2mb disk images.
I'm not a great fan of cloudflare - but they do some interesting work in bringing ipfs into the "normal" cloud-style static site/"pages"-hosting - either letting users expose cf pages via ipfs:
Unfortunately, as far as I understand, they closed down the open ipfs gateway [b] [p] - it's now up to "users" to pay for bandwidth exposed via ipfs on the web:
The most important part of IPFS is Content Based Addressing, not p2p. Content Based addressing provides a natural way for data to be portable, because it's not tied to a storage location or to a transport protocol.
And works great until your content changes, and then the address changes, and you end up wanting to have a way to name and address your content across the changes, and then you aren't using content addressing anymore.
the metadata can be versioned and you can lookup by version, so names of stored file can be changed but every time there is a change to any of the metadata the version is updated too
It works great even in that case, at the fundamental level it makes sense for content to be accessed in such a manner and then indexed on too of that by name.
Well, to me those sounds like different use cases. Pointing to the current up do date content requires a domain anyway. The domain is the users guarantee that a trusted party has made the decision to update the content.
An end-user doesn't want to manually update a domain each time they update their site. That's practically a non-starter for the technology. Especially since domain updates require transactions (on a L1 or L2 chain) to complete.
As mentioned by another person, IPNS does solve this issue, though. My domains are pointed to an IPFS+IPNS-published version of my site, which is re-published on change from Github.
I thought IPFS did both; default content-addressing but with IPNS to allow mutability if desired, which means you can use whatever works the best. Did I misunderstand that?
Because it's no-longer content addressing: if you are addressing your content by a name, to allow it to mutate, then you aren't addressing it by its content. You're addressing it by a name.
And not to beat a dead horse, but IPNS doesn't work very well. These days people tend to build their own naming systems using layers of IPLD hashes instead.
You're still using content addressing. Any mirrors made stay the same. Any changes are preserved as new versions. All versions are still accessible by their hashes.
The fact that you have some extra names on top of that doesn't change the base system.
Just use a public key as your address, and make signed pointers to the actual content. That's how people have been solving the update problem for ages.
Naming things continues to be one of the hard problems in computer science.
Why not? If you include the public key with the signed pointer, then you can define a "hash" function that returns just the pubkey (yes with collisions, but not in a way that matters) and it is content-addressing. After all, you are literally addressing by content in this scheme.
What is actually the difference in any way that matters? What property does content addressing have that this doesn't.
My point is that you don't need p2p to have content based addressing. You "only" need a way to discover a provider for a given hash. p2p is one way, but other network topologies are also interesting to improve performance, for instance mobile clients can benefit from trustless edge gateways.
I think IPFS fails at the most basic tenet of file storage which is file permanence. It's touted as a decentralized approach but then anyone whose anyone ends up paying a centralized service to "pin" their content.
Arweave is leagues better: I use it all the time to store articles I've written and very precious media. It couldn't be easier to use with its browser extension. It's also very cheap for true file permanence forever at least until all the storage nodes die out but given replication stands at some absurd multiplier there's little chance of it.
"Arweave is a protocol that allows you to store data permanently, sustainably, with a single upfront fee."
This is economically impossible. That is the description of a pyramid scheme, a startup without a revenue model trying to make up for the fact that they're losing money on every customer by getting lots of customers, a business major who skipped math class, or all of the above.
You can't support indefinite retention on fixed cost.
Miners are incentivized to store data via a storage endowment, which pays a principle up-front and accrues interest over time. [1] As cost of storage decreases faster than the interest, storage is secured. Read the yellow paper for a technical deep dive. [2]
Annuities that pay indefinitely are real financial products with finite prices. If I wanted to be more specific I maybe should have written "perpetuities."
Would you really price one dollar per year forever at infinity dollars today? That pricing implies some unusual beliefs about the time value of money.
Okay, if they're real financial products, I'd like to buy one. Which banks will sell me a commitment to pay out one dollar per year forever, including after I die?
I guess when someone uses a comment section as a substitute for a search engine I should not expect them to believe that any financial product that their bank will not sell them exists. https://en.wikipedia.org/wiki/Perpetual_bond#Examples
Indian banks are issuing these things in current_year. You can just buy them. Do you have an argument about why the formula used for pricing these bonds is incorrect? What do you think about a related financial instrument backed by sovereign bonds that are rolled at expiry? Is it also impossible?
I expect that the price of debt issued by Indian banks correctly incorporates the probability that those banks will continue to exist and honor that debt until the end of time, which is 0.
The part you are missing is that the entire system has to die for your specific data to be lost. That may happen, but it's a different risk than imagining that somebody is storing a single piece of data indefinitely for a single up front fee.
The risk and mispricing are socialized across the entire protocol into perpetuity.
Could you explain why VCs are involved in this token from an ICO?
I regard any crypto project, (including bitcoin) a scam especially if it is a VC pre-mined token.
There are other better technologies out there that already exists that works that doesn’t need to require the use a blockchain or have a token attached to it.
Your definition is emotionally-fueled and bears little relevance to the actual definition of a scam. I don't know any better technology that exists. The Internet Archive is supported by donations and subject to tides of political sentiment.
BitTorrent has a huge problem with data retention and is the only other decentralized popular alternative I know of that can be considered successful.
> Your definition is emotionally-fueled and bears little relevance to the actual definition of a scam.
This is a shallow dismissal without addressing any of my points of which they are facts [0].
What is certain is that this Arweave project is searching for a problem with blockchain and it doesn’t exist. Cramming blockchain into it doesn’t make it novel or better than what already exists.
You’ve said it yourself, BitTorrent & The Internet Archive works well after 25+ years.
More importantly it is not surprising that this coin is a premine, I would argue that most of the money from that ICO would have been better allocated to IA (a non profit) than to some ephemeral for profit crypto project, that doesn’t do anything than enrich A16Z, Coinbase and all the other early investors who clearly want to make a quick buck in crypto.
I would have expected HN users to be more sceptical of pyramid schemes like this one and it seems that you are either: invested in this token, or are extremely naive enough to think this project would legitimately work at scale (assuming regulations in the crypto space are implemented), or both.
IPFS never promised anyone they could just put their stuff out there and have it last forever. That's equivalent to free energy. That's the pitch for Filecoin, but not IPFS.
And as far as centralization, you've got a Pareto distribution, no network topology can change that.
The majority of seeding/downloading will be done with the (minority) most popular files, while only a minority of seeding/downloading is done for the majority of files.
For example, go to TPB and look at the distribution of seeders/leechers.
Ahhhh of course! I wonder if there is a game-theoretically-sound way to smooth out the distribution. At first glance it seems not, because chunks (probably not using the right term for torrent file partitions) need to be addressable/discoverable. However, what if there was a way to couple “popular” chunks with slightly less popular chunks, such that seeding becomes some what more equitable for, say, the top 30-40% of files
Then I believe it fails at what I'll call a philosophical pillar of storage, which is that file objects saved or distributed anywhere should be durable. I, personally, have no use for IPFS as every use case I have for file storage and distribution is solved by Arweave. You may have other needs than my own which are better solved by IPFS. If so, I'd love to hear what they are.
my problem with filecoin is that it is essentially disconnected from IPFS… it’s basically completely unrelated. And you need big contraptions to use it together.
Protocol Labs people always sell “Filecoin” as solution to IPFS shortcomings… but it isn’t, really. Filecoin is an entirely separate network where you pay for both retrieval and storage. And you need big contraptions to use those two together in any way. It isn’t made to be used together at all.
I think if you're really looking for permanence, the best bet is to make lots of copies using diverse storage systems -- including Filecoin, IPFS' own incentivized storage system, ArWeave, etc -- and then use a content-addressable protocol like IPFS as the front end to access them. That way, whatever system survives, you'll still be able to seamlessly locate and refer to your data.
(Full disclosure, I work for the Filecoin Foundation for the Decentralized Web, which is certainly part of the wider IPFS Cinematic Universe.)
Just like with normal files, you can pay a third party to keep them around or store them yourself. There are some cryptocurrency based cloud providers that will subsidise cloud storage for their content but like with the rest of the internet, someone will need to pay.
I wish there was a version of IPFS that distributed popular files as the swarm demands. Any singular person doesn't have whole pieces and doesn't know what they are distributing. Also when you help the swarm you get a crypto token of some sort, giving people a financial incentive to help the swarm.
So you would say I have 16TB, feel free to use it as you see fit swarm. I can "mine" crypto by helping the swarm.
The swarm benefits from a gigantic pool of storage and content.
ISPs and government agencies can't do anything because it's all anonymous and any individual person is sharing encrypted pieces of a file anyways.
> ISPs and government agencies can't do anything because it's all anonymous and any individual person is sharing encrypted pieces of a file anyways.
It would need to be over Tor or similar, which is of course way slower. Otherwise there's nothing preventing "ISPs and government agencies" from going after people sharing copyrighted bits.
People keep mentioning the financial incentive as either something to take issue with and combat the entire system, or as something to hold up as a feature which can save a project.
Perhaps both have some point; however, I don't see it really necessary at all for a very useful and thriving project to benefit from something like IPFS.
Case in point - DVC.
You don't need financial incentive for people to store ML models / weights on their computers - they need to use them there anyway. It makes great sense that the more popular the model, the faster it should load from more nodes serving you, rather than waiting for things to be downloaded from the one server the authors put it on (that will probably be deleted in a year or so anyway).
> ISPs and government agencies can't do anything because it's all anonymous and any individual person is sharing encrypted pieces of a file anyways.
Note that there is no (happy to be corrected) publicly available IPFS implementation that can be run reasonably anonymous. Trying to tunnel kubo (go-ipfs) over Tor or other proxies will break stuff. There's also a bunch of fingerprinting that is not possible to disable without modifying source.
The answer to HA IPFS seems to be "just run more nodes". I am yet to speak with someone who has tried and reached success with load-balancing IPFS using available load balancers.
Those who said that do not know the torrent protocol which is why the onion project even had to issue a statement on why using torrents over Tor is not secure.
>So you would say I have 16TB, feel free to use it as you see fit swarm. I can "mine" crypto by helping the swarm.
The problem is that cryptocurrency mining is about producing verifiable proofs. I don't know how you would go about producing a proof that you've helped the swarm in a decentralized, sybil-proof way.
I love IPFS for some things. It's sort of not anonymous and it's nondeterministically slow in starting and keeping itself going. When it works, it's fantastic. I think an overlay fs on top of bittorrent (and adopting existing torrents) would've been more useful than reinventing the wheel and fragmenting. Torrents already have CBA and work.
It also has the problem of not enough people use it.
It has some performance and reliability problems. Some of them are just a matter of writing a better implementation, some are more conceptual.
But the basic principle of a content-addressed, distributed file system is desperately needed, and there is a lot of work going on behind the scenes from both protocol labs and others to fix these problems...
I like the "immutability" of IPFS. Back in 2017 I tried it out and [Dat](https://www.datprotocol.com/). Both IPNS and Dat seemed to break often (not finding the correct peer, which then lead to broken downloads).
With IPFS, you know "it's just a hash of the content". That idea of content addressing is pretty powerful, and I'm starting to see it a lot more, particularly in the NixOS ecosystem.
IPNS, on the other hand, seldom worked. The 24hr timeout made it unusable without an abstraction on top that kept it up to date. Dat had a similar problem, one of storing way too much historical data, and I could never be sure if the peers had the same version I las published.
I think these are "theoretical problems". Content addressing is a great idea and it's reliable. Distributed hash tables are on the weaker technology spectrum. I'm rooting for IPFS to replace bittorrent, not IPNS/Hypercore, which I think will have serious scalability/security /privacy problems.
The problem with torrents is a lack of expanding/versioning of a torrent
e.g. I'm Joe Rogan and I release all my podcasts into a single torrent, add one each podcast. JoeRoganPodcastArchive.torrent . Each addition should propagate if the torrent client accepts it (i.e. Joe can't sneak in a episode of Oprah as the user would catch that).
Debian_amd64_ISO_Archive.torrent , each release Debian chuck it in there. Up to the user to accept the new addition (likewise download selectively, as can be done already).
By now you should get the drift of what I'm on about. The key is you don't have to get another torrent file to "follow" (what would be) the expanding torrent.
It also serves to propagate new content faster, the change comes to the user rather than the user having to seek it out.
RSS feeds currently fill this role and honestly I don't feel the need for base torrent protocol to support it. It does its job and it does it well, same as RSS.
Yes, clearly. I just was amused that that dipshit of all people was your go to example.
That sort of thing in a totally unrelated sort of comment tends to be a huge dogwhistle.
But I clearly just want you to “discriminate” against the wealthy middle aged male white asshole. (Disclosure, I’m a middle class middle aged white male asshole)
I did try to use ipfs but not having a search capability (something like torrent) is such a let down for the end user. I dont think it will stick just for this reason
I would absolutely love an IPFS based version control system, something superficially (or more) resembling fossil would be amazing. I think it's probably one of the best applications for IPFS, since a lot of the core data structures and primatives required for version control are already a part of IPFS. A version control system built on top of a distributed content addressed filesystem, what could be better?
A version control system with atrocious object retrieval performance, and with which you have pin everything yourself (or pay someone to pin everything) anyway to guarantee that all your stuff aren’t lost? No thanks.
You have to have files on your system anyway to work on or use the files. How is it any worse to have it pinned? And if everyone working on a project keeps a copy of the canonical branch there's no central place you'd need to look for the files, just query for the hashes of the files. Github and the like are quick fixes for the shortcomings of things like git and mailing lists, I think IPFS could help actually resolve those shortcomings.
The Blockchain is almost as old as the smart phone. By now, given that not one single application, aside from cryptocurrencies, has appeared, it's very reasonable to be skeptical.
If you could actually pay once to have a file stored forever, that would be useful. "Perpetual care" for data would be valuable. There are Filecoin people who will take your money to do that, but don't have the credibility.
Now, if Berenberg Bank (private bankers since 1590 and still a substantial bank) managed the funding for this, it might be credible. One of the things they do is handle money on the scale of centuries.
Actually, I believe that BitTorrent comes pretty close. The magnet links are just the SHA-256 hash of the content (in truncated). Anyone can link to such content and as long as at least one computer worldwide has the data stored and is online, you can still download it. Modern BitTorrent doesn't even need real trackers anymore, due to its distributed hash table of peers.
IPFS doesn't solve permanent data availability either, the closest to "solving" this is Arweave[1] but that comes with problems of its own. It's fundamentally impossible to correctly price the availability of a "service" in the far future.
IPFS works just like BitTorrent in this respect. The main differences are that IPFS has more affordances for serving files to browsers so websites could be hosted through it through a standard IPFS web gateway, and that Filecoin (a system for paying people to verifiably seed your content) integrates with it.
the problem with bittorrent is, if you seed n torrents at the same time. Thats n different pools to keep up to, n dhts to manage, and so on.. With ipfs you seed blocks (folders,files,content) and it does not matter how much, the inital overhead stays the same.
e.g. apt could never use torrents due to the amount of small packages.
> TLDR: people whose salary depends on selling IPFS think everyone needs IPFS
Kind of important to point out this isn't in and of itself saying they are incorrect or lying. A carrot farmer may promote the health benefit of vegetables without nefarious intent.
They aren't strictly incorrect or lying. They just do a very bad job at doing a thorough and honest analysis of the history, literature & software landscape of data transfer and file servers. Which is to say this is a motivated public communication and not some scientific/engineering analysis, prognostic and recommendations.
It's weird that IEEE allows these type of "almost paid promotions". When I see the IEEE domain I automatically have a high degree of trust, but stuff like this will change my perception...
I love the concept of IPFS but every time I try to use it I run into very practical problems. The IPFS server takes up tons of CPU time seemingly doing nothing. Requesting a file from any public gateway to test availability takes minutes even for small files.
I like the design concept of the main software (a drag and drop sharing application that happens to enable IPFS rather than some kind of specific server/client kit) but there needs to be better server software for this to ever be popular. You need IPNS at the very least to have a website equivalent and updating that automatically requires some custom tooling already.
Then there's the privacy problem. In the default settings, you can query what IP read what content. You can run IPFS over Tor but that requires command line knowledge at the very least. The docs briefly mention it but don't go into detail.
The NFT boom brought more attention to IPFS and I don't think that's necessarily a good thing; I don't think the association with cryptocurrency scams and IPFS will help the protocol forward.
I like IPFS a lot and I'm very interested in seeing how the technology develops, but I'll note that this is written by ProtocolLabs, the founder of IPFS.
Peertube is based on a related, but perhaps more practical idea. In Peertube, someone has the master copy, and watching nodes also cache the content and redistribute it. It's not distributed storage, just distributed caching. You can post a viral video on a small site without being overwhelmed by traffic demands. It's proper streaming, not just file hosting; you can jump around within the video. Most importantly, no annoying ads.
It does an OK job. I put technical videos and rendering tests on Peertube.[1]
If you don't need "discovery" or "monetization", it's a good option.
160 comments
[ 3.0 ms ] story [ 217 ms ] threadCare to explain why? I only know the basic idea of IPFS and sounded great to me and something I was looking forward to get stable and widely adopted.
In which way is it a joke?
If you want something to be fast then your content needs to be hosted on servers with fast connectivity to other servers. And that's not possible when at the same time you're trying to avoid legal jurisdictions that will censor e.g. EU, US, CN etc.
Which is to say - the main technical problems that hampered BitTorrent probably aren’t there anymore.
It is also probably (I'm not quite sure yet) worse for the illegal uses because it is slower and more compute heavy than torrents, and there is way worse discoverability and moderation, which is important in the communities sharing illegal files.
Basically any true decentralisation is an attempt to eliminate trust from the system. And there are two problems with this - 1) maybe 0.1% of all attempts are actually decentralised (other 99.9% are either self delusional or lying), and 2) trust is actually a desirable thing, especially in the important things like money or file storage.
But if you run just a node, everything that is on the node is there because either you have added it or you have explicitly requested it.
IPFS is not like freenet or other systems that shard data over all participants.
I'm not sure how you are imagining IPFS works.
This obviously hasn't been the case for bittorrent, so I don't know why you would assume that would be the case for IPFS. Even ignoring the legal risk of blindly re-hosting whatever illegal bootleg-ISIS-child-porn-malware.exe on the net, the use of a blacklist isn't sybil proof so you just get bogged down with spam by the first attacker and run out of disk space.
You don't add them manually, its just when you open up a link in your browser or video player, it gets stored for some amount of time and seeded before perhaps being pruned or kept forever depending on the users preference. You don't have any issues with unknowingly hosting illegal files this way.
One area where IPFS improves on HTTP is in allowing comprehensive auditing of stored data. For example, thanks to its content-addressing functionality and, in particular, to the use of unique and permanent content identifiers, IPFS makes it easier to determine whether certain content is present on the network, and which nodes are storing it. Moreover, IPFS makes it trivial for users to decide what content they distribute and what content they stop distributing (by merely deleting it from their machines)."
p2p makes most sense for ephemeral transactions.
There are services where I don't want them going down ever of course. But those are more critical infrastructure than something like Apple Music or YouTube.
Dont get me wrong I like IPFS and would love to see increased usage but if a libgen mirror is the biggest example then we need to find more applications for it.
Unlike with torrents, most people will be using the “gateways” - most people are not “seeding” - partly because IPFS itself is resource hungry and hard to use, unlike Torrent which is easy to use.
In reality IPFS world is very centralized, you just have this resource hungry pretend-decentralisation on top.
I don’t know why is it like that when BitTorrent is much older and much more actually decentralised.
The way BitTorrent fixed this was the ratio - if you weren't seeding enough, you'd eventually get kicked out from the tracker. Does IPFS have a similar mechanism, or something else but with the same purpose?
At my previous company actyx.com we wrote our own ipfs implementation in rust called ipfs-embed ( https://github.com/ipfs-rust/ipfs-embed ) that serves as the storage for our distributed event sourcing system. It is used in production and is reliable.
The user of an Actyx deployment will not notice in any way that they are using ipfs in the background.
There are also some other interesting developments involving ipfs and rust. See a list of implementations here:
https://docs.ipfs.tech/basics/ipfs-implementations/
Here is my talk about actyx from IPFS camp 2019: https://www.youtube.com/watch?v=Qzu0xtCT-R0 . It does not go into much technical details, and it does not describe the current architecture but a previous iteration though.
Bitswap has a compat mode, but when two ipfs-embed versions talk to each other they use a different protocol.
Note that ipfs-embed is made for a different use case than the global ipfs. It is typically used by private swarms where there are a smaller number of nodes. So e.g. we don't use the DHT at all.
We also found that go-ipfs compat is not that useful for our use case, so the compat layer was mainly done as a service to the community and might not work with the latest go-ipfs / kubo.
Basically the nodes of the swarm (around 100 at most) gossip events to each other. That is also how they learn about the presence of the other nodes. They try to stay connected as well as the network topology allows.
Then when you want some bulk data via bitswap they just ask their current peers. Either the emitter of the data is a current peer, then it obviously has it. Or eventually one of your peers will replicate the data, and then you will get it with some delay.
The latter happens frequently if a part of the swarm is behind a NAT. You have some delay, but eventually everybody gets all events.
https://winworldpc.com/download/7f8040e5-d206-11e7-a73f-fa16...
https://blog.cloudflare.com/cloudflare-pages-on-ipfs/
Unfortunately, as far as I understand, they closed down the open ipfs gateway [b] [p] - it's now up to "users" to pay for bandwidth exposed via ipfs on the web:
https://blog.cloudflare.com/ea-web3-gateways/
[b] https://blog.cloudflare.com/distributed-web-gateway/
[p] https://cloudflare-ipfs.com/
As mentioned by another person, IPNS does solve this issue, though. My domains are pointed to an IPFS+IPNS-published version of my site, which is re-published on change from Github.
And not to beat a dead horse, but IPNS doesn't work very well. These days people tend to build their own naming systems using layers of IPLD hashes instead.
The fact that you have some extra names on top of that doesn't change the base system.
If your address is a function of a public key, then you're using "agent key addressing."
Naming things continues to be one of the hard problems in computer science.
I think that's a fine road to go down for addressing data, but it's not content addressing.
What is actually the difference in any way that matters? What property does content addressing have that this doesn't.
Also not like this was an ipfs innovation. Magnet links (bit torrent and predessor file sharing p2p), freenet, etc all significantly predate ipfs
Arweave is leagues better: I use it all the time to store articles I've written and very precious media. It couldn't be easier to use with its browser extension. It's also very cheap for true file permanence forever at least until all the storage nodes die out but given replication stands at some absurd multiplier there's little chance of it.
https://arwiki.wiki/#/en/Arweave
This is how they describe their project:
"Arweave is a protocol that allows you to store data permanently, sustainably, with a single upfront fee."
This is economically impossible. That is the description of a pyramid scheme, a startup without a revenue model trying to make up for the fact that they're losing money on every customer by getting lots of customers, a business major who skipped math class, or all of the above.
You can't support indefinite retention on fixed cost.
[1] https://www.arweave.org/technology#blockweaves [2] https://www.arweave.org/yellow-paper.pdf
If so, then yes, this system depends on an infinite stream of new customers paying in in order to continue to support the existing ones.
Would you really price one dollar per year forever at infinity dollars today? That pricing implies some unusual beliefs about the time value of money.
More to the point, an organization can write a promise on a piece of paper. That does not make it economically possible to keep that promise.
The risk and mispricing are socialized across the entire protocol into perpetuity.
I regard any crypto project, (including bitcoin) a scam especially if it is a VC pre-mined token.
There are other better technologies out there that already exists that works that doesn’t need to require the use a blockchain or have a token attached to it.
BitTorrent has a huge problem with data retention and is the only other decentralized popular alternative I know of that can be considered successful.
This is a shallow dismissal without addressing any of my points of which they are facts [0].
What is certain is that this Arweave project is searching for a problem with blockchain and it doesn’t exist. Cramming blockchain into it doesn’t make it novel or better than what already exists.
You’ve said it yourself, BitTorrent & The Internet Archive works well after 25+ years.
More importantly it is not surprising that this coin is a premine, I would argue that most of the money from that ICO would have been better allocated to IA (a non profit) than to some ephemeral for profit crypto project, that doesn’t do anything than enrich A16Z, Coinbase and all the other early investors who clearly want to make a quick buck in crypto.
I would have expected HN users to be more sceptical of pyramid schemes like this one and it seems that you are either: invested in this token, or are extremely naive enough to think this project would legitimately work at scale (assuming regulations in the crypto space are implemented), or both.
[0] https://icodrops.com/arweave/
And as far as centralization, you've got a Pareto distribution, no network topology can change that.
Explain for the more obtuse among us?
For example, go to TPB and look at the distribution of seeders/leechers.
https://www.youtube.com/watch?v=fCn8zs912OE
Protocol Labs people always sell “Filecoin” as solution to IPFS shortcomings… but it isn’t, really. Filecoin is an entirely separate network where you pay for both retrieval and storage. And you need big contraptions to use those two together in any way. It isn’t made to be used together at all.
(Full disclosure, I work for the Filecoin Foundation for the Decentralized Web, which is certainly part of the wider IPFS Cinematic Universe.)
So you would say I have 16TB, feel free to use it as you see fit swarm. I can "mine" crypto by helping the swarm.
The swarm benefits from a gigantic pool of storage and content.
ISPs and government agencies can't do anything because it's all anonymous and any individual person is sharing encrypted pieces of a file anyways.
It would need to be over Tor or similar, which is of course way slower. Otherwise there's nothing preventing "ISPs and government agencies" from going after people sharing copyrighted bits.
Perhaps both have some point; however, I don't see it really necessary at all for a very useful and thriving project to benefit from something like IPFS.
Case in point - DVC.
You don't need financial incentive for people to store ML models / weights on their computers - they need to use them there anyway. It makes great sense that the more popular the model, the faster it should load from more nodes serving you, rather than waiting for things to be downloaded from the one server the authors put it on (that will probably be deleted in a year or so anyway).
Note that there is no (happy to be corrected) publicly available IPFS implementation that can be run reasonably anonymous. Trying to tunnel kubo (go-ipfs) over Tor or other proxies will break stuff. There's also a bunch of fingerprinting that is not possible to disable without modifying source.
The answer to HA IPFS seems to be "just run more nodes". I am yet to speak with someone who has tried and reached success with load-balancing IPFS using available load balancers.
Same thing has been said about BitTorrent and Tor.
And yet governments have been able to de-anonymize file sharers.
https://blog.torproject.org/bittorrent-over-tor-isnt-good-id...
The problem is that cryptocurrency mining is about producing verifiable proofs. I don't know how you would go about producing a proof that you've helped the swarm in a decentralized, sybil-proof way.
https://www.ethswarm.org
(Disclosure: I am working on the project)
It also has the problem of not enough people use it.
But the basic principle of a content-addressed, distributed file system is desperately needed, and there is a lot of work going on behind the scenes from both protocol labs and others to fix these problems...
Stay tuned :-)
With IPFS, you know "it's just a hash of the content". That idea of content addressing is pretty powerful, and I'm starting to see it a lot more, particularly in the NixOS ecosystem.
IPNS, on the other hand, seldom worked. The 24hr timeout made it unusable without an abstraction on top that kept it up to date. Dat had a similar problem, one of storing way too much historical data, and I could never be sure if the peers had the same version I las published.
I think these are "theoretical problems". Content addressing is a great idea and it's reliable. Distributed hash tables are on the weaker technology spectrum. I'm rooting for IPFS to replace bittorrent, not IPNS/Hypercore, which I think will have serious scalability/security /privacy problems.
[1]: https://www.bittorrent.org/beps/bep_0046.html
e.g. I'm Joe Rogan and I release all my podcasts into a single torrent, add one each podcast. JoeRoganPodcastArchive.torrent . Each addition should propagate if the torrent client accepts it (i.e. Joe can't sneak in a episode of Oprah as the user would catch that).
Debian_amd64_ISO_Archive.torrent , each release Debian chuck it in there. Up to the user to accept the new addition (likewise download selectively, as can be done already).
By now you should get the drift of what I'm on about. The key is you don't have to get another torrent file to "follow" (what would be) the expanding torrent.
It also serves to propagate new content faster, the change comes to the user rather than the user having to seek it out.
That sort of thing in a totally unrelated sort of comment tends to be a huge dogwhistle.
But I clearly just want you to “discriminate” against the wealthy middle aged male white asshole. (Disclosure, I’m a middle class middle aged white male asshole)
I mean it's their lives but I like their work, except the silly crypto stuff.
Also HN people: Stop wasting time trying to make legitimate use cases for crypto!
I don’t see the issue here.
You disprove "no use cases" by finding one where a blockchain is a natural fit. Not by taking a standalone idea and grafting a blockchain onto it.
Heck, IPFS is over seven years old!
One is paid pinning services - services that ensure the existence and availability of files in the network.
Then there i FileCoin - an augmentation with a specialised economy that tried to have files stored an available based on a market.
On the other hand, bittorrent would probably benefit from a FileCoin like economy build on top of it.
TLDR: people whose salary depends on selling IPFS think everyone needs IPFS
If you could actually pay once to have a file stored forever, that would be useful. "Perpetual care" for data would be valuable. There are Filecoin people who will take your money to do that, but don't have the credibility.
Now, if Berenberg Bank (private bankers since 1590 and still a substantial bank) managed the funding for this, it might be credible. One of the things they do is handle money on the scale of centuries.
[1]: https://www.arweave.org/technology
Kind of important to point out this isn't in and of itself saying they are incorrect or lying. A carrot farmer may promote the health benefit of vegetables without nefarious intent.
I like the design concept of the main software (a drag and drop sharing application that happens to enable IPFS rather than some kind of specific server/client kit) but there needs to be better server software for this to ever be popular. You need IPNS at the very least to have a website equivalent and updating that automatically requires some custom tooling already.
Then there's the privacy problem. In the default settings, you can query what IP read what content. You can run IPFS over Tor but that requires command line knowledge at the very least. The docs briefly mention it but don't go into detail.
The NFT boom brought more attention to IPFS and I don't think that's necessarily a good thing; I don't think the association with cryptocurrency scams and IPFS will help the protocol forward.
It does an OK job. I put technical videos and rendering tests on Peertube.[1]
If you don't need "discovery" or "monetization", it's a good option.
[1] https://video.hardlimit.com/w/sFPkECUxRUSxbKXRkCmjJK