Is IPFS mature and reliable? Genuine question. I've always been curious about, but never had a need for it. Since it didn't take over the world as we know it, I believe something must be missing yet.
The main thing IPFS does better (in my opinion) is that the user interface is based around "browsing web sites" instead of "downloading files". For example, you can upload all of Wikipedia to IPFS such that users can easily read specific articles they're interested in without downloading the whole thing.
I think this is mostly a UI thing, not an improvement to the underlying technology. IIRC it's technically possible to "stream" from torrents or download chunks of them, it's just not the default way people interact with them.
IPFS is a HTTP replacement much more than it is a BT replacement. Meaning it provides namespace to access and link content, but unlike HTTP that is build around server addresses, IPFS build around content addressing. Another big advantage IPFS has over HTTP is that it supports directories natively, thus making the use of archive files largely unnecessary (in theory it could also be extended to address files inside archives directly, so that's not implemented I think).
In theory that allows some extremely powerful things, such as making all the content of the world part of your filesystem. You can just "cd /ipfs/<long hash>/" and browse around whatever directory tree you are interested in. It completely bypasses the need to manually download anything, as you can access the data directly under a globally unique name.
IPFS is also much more suited for small files, so you could use it to host individual .jpg files, something that would be rather unwieldy with torrent.
In actual use however it barely works, it's slow, it's eats lots of CPU even when doing nothing and the fuse plugin to integrate it into your filesystem is also rather broken. POSIX file io is also ill suited for this kind of transparent file access, so all your apps will just freeze without any indication what is going on if you try to use the /ipfs/ directory.
Another huge problem with IPFS is the complete lack of authorship/copyright tracking. HTTP has that implicitly, since whoever hosts the content is responsible for it. With something like IPFS however you automatically become hoster for everything you look at. I can't see that working in the current legal system. Even distributing Open Source projects via IPFS is already illegal, as the LICENSE.txt you are generally required to redistribute no longer is linked with the actual rest of the source code once you can address and cache each file, or even blocks within a file, individually. If IPFS would ever become popular it would be quickly made unusable by lawyers, as unlike torrent, you downloading is not an active act here, it just happens in the background.
IPFS promises a lot of things (content-addressable, unique identifiers, fast, efficient, durable) but hasn't delivered on any of them. It's basically a slower + more complicated reimplementation of Bittorrent's magnet links.
It's extremely useful for publishing and downloading ML models. Huggingface and such are not going to be reliable, so it is a smart move to use academic torrents or ipfs instead.
There's a radius from earth's surface where TCP/IP won't work anymore (speed of light and whatnot). So you need some kind of datagram protocol. Also if you need Earth's infrastructure like DNS on Mars when Sol is in between Mars and Earth -- well, good luck I guess!
Delay/Disruption Tolerant Networking is an attempt at some of this to actually make an interplanetary protocol. Delayed routing protocols appear to be super hard -- not to mention the problem with issues with MITM attacks at the relays.
There's also interesting aspects of the space application of DTN where transmitters needing to have knowledge of solar system orbits (including the relays) to broadcast at the right time so the message will arrive when the relay is on the right side of Mars, say.
It's looks like LunaNet is going to be the solution for the Artemis mission. But the moon is only 1.3 light seconds away, so not really a super hard challenge (except when trying to transmit to the side opposite the earth). Heck, TCP will probably still work through purely analog relays, albeit terribly.
But I'm more interested in what happens for Mars missions when the round trip time is going to be measured in minutes.
The APID in that spec (88 bits) is where a lot of complexity (imo) is pushed up to the application layer and seems key to the archicture. It looks like basically a typed-data+endpoints addressing scheme and assigned to endpoints per mission by that mission itself.
> > Delay/Disruption Tolerant Networking
> How interesting. (Surprised this didn't come up in IoT domain.)
The term may not be commonly used there, but like everything in tech, people often try to solve the same problems from different angles using different terms. Offline First is also roughly comparable to DTN and some people in the IPFS have framed it in the context of Offline First.
DTN can actually solve some of the most pressing communications issues we face today: 40% of the world population is permanently disconnected from the Internet, and many of those who are are subject to severe censorship (look up #KeepItOn).
I don’t know this spec but my skimming the DTN webpage and the SPP gives me the impression that this architecture is in part designed to insure end-to-end established logical circuits, so in context of [censorship/access-denial] I don’t know if it is a good fit.
NASA’s HDTN [1] says it is based on IETF RFC 5050 [2]. Have to read that to understand privacy and censorship implications, specially the Bundle Security Protocol [3]. But just skimming the index in those specs reinforces my impression that this architecture can lockdown/lockout at administrative points.
The stressed environment of the underlying networks over which the Bundle Protocol will operate makes it important for the DTN to be protected from unauthorized use, and this stressed environment poses unique challenges for the mechanisms needed to secure the Bundle Protocol.
Furthermore, DTNs may very likely been deployed in environments where a portion of the network might become compromised, posing the usual security challenges related to confidentiality, integrity, and availability.
-- end --
& p.s. RFC 4838 mentions "optional" security protocols (such as rfc 5050). There is also a "registration" protocol (to allow connection continuity over process restarts) "that may fail".
I have to think about this but atm not 100% whether these requirements are inherent to the problem domain (overlay, mix of networks, partitions, ..)
> But I'm more interested in what happens for Mars missions when the round trip time is going to be measured in minutes.
Same here.
This is going to come across as a shameless plug, but implementing DTN tech is actually my full-time job and we've got a couple of projects that will address some of the issues you identified:
- Awala, a computer network where compatible apps can use the 'standard' Internet infrastructure when it's available, or fall back to a backup medium that tolerates delays of up to 6 months. https://awala.network/
- VeraId, an offline authentication protocol powered by DNSSEC. https://veraid.net/
16 comments
[ 3.1 ms ] story [ 51.2 ms ] threadI think this is mostly a UI thing, not an improvement to the underlying technology. IIRC it's technically possible to "stream" from torrents or download chunks of them, it's just not the default way people interact with them.
In theory that allows some extremely powerful things, such as making all the content of the world part of your filesystem. You can just "cd /ipfs/<long hash>/" and browse around whatever directory tree you are interested in. It completely bypasses the need to manually download anything, as you can access the data directly under a globally unique name.
IPFS is also much more suited for small files, so you could use it to host individual .jpg files, something that would be rather unwieldy with torrent.
In actual use however it barely works, it's slow, it's eats lots of CPU even when doing nothing and the fuse plugin to integrate it into your filesystem is also rather broken. POSIX file io is also ill suited for this kind of transparent file access, so all your apps will just freeze without any indication what is going on if you try to use the /ipfs/ directory.
Another huge problem with IPFS is the complete lack of authorship/copyright tracking. HTTP has that implicitly, since whoever hosts the content is responsible for it. With something like IPFS however you automatically become hoster for everything you look at. I can't see that working in the current legal system. Even distributing Open Source projects via IPFS is already illegal, as the LICENSE.txt you are generally required to redistribute no longer is linked with the actual rest of the source code once you can address and cache each file, or even blocks within a file, individually. If IPFS would ever become popular it would be quickly made unusable by lawyers, as unlike torrent, you downloading is not an active act here, it just happens in the background.
>Since it didn't take over the world as we know it, I believe something must be missing yet.
Not sure that's how it works, good ideas get buried all the time.
Seems like it's momentum is gone and if you haven't interacted with it yet you probably shouldn't bother.
Delay/Disruption Tolerant Networking is an attempt at some of this to actually make an interplanetary protocol. Delayed routing protocols appear to be super hard -- not to mention the problem with issues with MITM attacks at the relays.
There's also interesting aspects of the space application of DTN where transmitters needing to have knowledge of solar system orbits (including the relays) to broadcast at the right time so the message will arrive when the relay is on the right side of Mars, say.
It's looks like LunaNet is going to be the solution for the Artemis mission. But the moon is only 1.3 light seconds away, so not really a super hard challenge (except when trying to transmit to the side opposite the earth). Heck, TCP will probably still work through purely analog relays, albeit terribly.
But I'm more interested in what happens for Mars missions when the round trip time is going to be measured in minutes.
How interesting. (Surprised this didn't come up in IoT domain.)
NASA's DTN: https://www.nasa.gov/directorates/heo/scan/engineering/techn...
& there is something called Space Packet Protocol: https://public.ccsds.org/Pubs/133x0b2e1.pdf
The APID in that spec (88 bits) is where a lot of complexity (imo) is pushed up to the application layer and seems key to the archicture. It looks like basically a typed-data+endpoints addressing scheme and assigned to endpoints per mission by that mission itself.
The term may not be commonly used there, but like everything in tech, people often try to solve the same problems from different angles using different terms. Offline First is also roughly comparable to DTN and some people in the IPFS have framed it in the context of Offline First.
DTN can actually solve some of the most pressing communications issues we face today: 40% of the world population is permanently disconnected from the Internet, and many of those who are are subject to severe censorship (look up #KeepItOn).
NASA’s HDTN [1] says it is based on IETF RFC 5050 [2]. Have to read that to understand privacy and censorship implications, specially the Bundle Security Protocol [3]. But just skimming the index in those specs reinforces my impression that this architecture can lockdown/lockout at administrative points.
[1]: https://github.com/nasa/HDTN
[2]: https://datatracker.ietf.org/doc/rfc5050/
[3]: https://datatracker.ietf.org/doc/html/rfc6257
—- from RFC 6257 —-
The stressed environment of the underlying networks over which the Bundle Protocol will operate makes it important for the DTN to be protected from unauthorized use, and this stressed environment poses unique challenges for the mechanisms needed to secure the Bundle Protocol.
Furthermore, DTNs may very likely been deployed in environments where a portion of the network might become compromised, posing the usual security challenges related to confidentiality, integrity, and availability.
-- end --
& p.s. RFC 4838 mentions "optional" security protocols (such as rfc 5050). There is also a "registration" protocol (to allow connection continuity over process restarts) "that may fail".
I have to think about this but atm not 100% whether these requirements are inherent to the problem domain (overlay, mix of networks, partitions, ..)
Same here.
This is going to come across as a shameless plug, but implementing DTN tech is actually my full-time job and we've got a couple of projects that will address some of the issues you identified:
- Awala, a computer network where compatible apps can use the 'standard' Internet infrastructure when it's available, or fall back to a backup medium that tolerates delays of up to 6 months. https://awala.network/ - VeraId, an offline authentication protocol powered by DNSSEC. https://veraid.net/