I've never tried that. Would you be willing to post the extended unbound stats, just the median group of latency timing? Here is one of my unbound nodes that does DoT to other DoT nodes I control that then do more DoT to other resolvers. I use a cron job to pre-cache most things I use so the latency isn't a big deal. I'm just curious what yours looks like. If you don't have it set to go to logs then it can be pulled with "unbound-control stats_noreset|grep histo"
With ODoH and DNSSEC this risk is mostly mitigated. It's a shame DNSSEC is often missing client implementation (and that the .com sphere barely implements it at all) but the risks have been mitigated pretty well in my opinion.
> the gateway is in full control and can inspect and tamper with the data as much as it likes
Inspect, yes, but tamper? IPFS content IDs are hashes, so it seems like it should be possible to verify what you have been sent if you really cared enough to do so.
Possible, yes, but probably the most widely used IPFS implementation – the Brave browser – does not implement this yet. They say they plan to in the future.
The merkle root of the DAG over the chunks is not related to the file's hash.
Imagine if I gave you a file out of a git repo, and the long commit hash. It's insufficient on its own to verify integrity, you need the intermediate nodes.
With ipfs you can infer that it used the reference chunker and try to reproduce the DAG but there is no guarantee it will work.
Better to just speak ipfs to the p2p net and get the DAG.
That is a single file, but good luck verify that file without actually downloading the entire dag tree which includes the file.
Worse, that same file can be chucked and bucketed infinitely number of ways. So if I download that file and the re-add it to IPFS, I will very likely get a _different_ address. And that address will likely share _none_ of the of the nodes from the previous address. Meaning even though the same file is in IPFS, they are completely distinct and don't actually help seed each other.
So basically CURL will get support for IPFS by using a proxy service (which is bad for user privacy).
And said proxy could bounce your request around for max 30 hops before a random IPFS-proxy honors the request. This is horrible for user privacy.
Brave browser [1] at least has a native support for IPFS and no need for proxies at all.
> Other IPFS privacy concerns: Brave, the browser, has actual native IPFS support (not using any gateway) and they have an informative page called How does IPFS Impact my Privacy?
This blog post about cURL is saying that they will not do this, and will require a manual configuration of a gateway.
> By default, Brave will load the URI being requested via a public HTTP gateway; however, it will also show an infobar that asks you if you’d like to use a local node to resolve IPFS URIs.
> And said proxy could bounce your request around for max 30 hops before a random IPFS-proxy honors the request. This is horrible for user privacy.
It literally says: I have insisted, in the PR for ipfs support to curl, that the IPFS URL handling code should not automatically follow such gateway redirects, as I believe that adds even more risk to the user so if a user wants to allow this operation, it should be opt-in.
Max redirs in curl defaults to 50. Most browsers default to around 20. 30X (in the article) is a range of status response codes. You would probably benefit by rereading the article.
This is silly. You could say the same for files CURLd from servers over http today. Did you know that when you download files from the internet, you're putting yourself at risk of the server tampering with your file before serving it to you??
The point of IPFS is that the server can't tamper with the content because it would break the hash, but if you're using an untrusted proxy then it is just as bad as using an untrusted proxy for your normal web browsing.
The server giving you the wrong file is not "tampering". The word refers to something in the middle making changes.
For the web, TLS provides tamper-resistance. Accessing the web through a HTTP proxy would allow tampering, similarly to accessing IPFS through a gateway.
It needs censorship-resistance added. It needs user privacy added.
Both of those could be enabled by combining data blocks semi-randomly. Ie. rather than downloading the data you're looking for, you download a block of data containing what you're looking for and some other data. Then nobody knows what it was you were looking at. Some combined blocks might contain a few bytes of data from thousands of files.
For censorship resistance, rather than users pinning individual files, the network needs to assign them blocks they must store (or be kicked off the network). Some of those blocks will be combined blocks which contain a few bytes some users may not wish to store - and that's fine, it means IPFS isn't for them.
Remember that data can be encrypted by content hash, so there is no way to know what you're storing unless you go scouring the public internet for a key to decrypt it.
> you download a block of data containing what you're looking for and some other data
That IPFS does NOT put data I did not explicitly requested on my storage is exactly why I feel comfortable using it compared to some other distributed storage systems where I might end up with horrific data on my system.
And encrypted data you don't have the key for is indistinguishable from random noise. 'Combined' blocks that you don't know which partner blocks are necessary for decoding are likewise useless.
You're being like a mailman handling mail - you can't look into the mail, so you can't know if you're delivering love letters, junkmail, or terrorist propaganda.
While true, if the government downloads objectionable/illegal data and they detect that 10% of that data is coming from your IP address you'll likely have a very bad day. Sure weeks or months later you might convince a judge and/or jury that you had no access or control to that data, and you might even get your laptops, desktops, and external storage back.
ISPs and USPS have common carrier protection, by default a home user does not.
This is such a key point that people forget: this isn’t some academic game theoretical debate but rather a question of how your actions will be perceived by society. If your IP is used to serve illegal content, saying “well actually, …” isn’t a defense and it most likely means you’re getting charged as an accessory to whatever that stranger did. Even if you can avoid legal penalties after years and significant expense, your reputation will be linked.
It depends on exactly what you’re talking about and which laws apply, but in general the problem for things like this, running a Tor exit node, etc. is that the content is something which people keep secret and the scenarios you need to worry about are related to serious crimes. If it’s, say, a small amount of copyrighted material your ISP will get an automated cease and desist letter but if your IP serves part of the requests for CSAM or non-consensual pornography, classified material, terrorism-related stuff, etc. that’s a very different situation. You might eventually be able to get off in court but that’s years after you’ve been in the news and quite likely had all of your devices confiscated in an unannounced raid (criminals have set up systems to destroy evidence so this isn’t without some justification), and you’d have incurred significant legal costs even if this didn’t cause you to become unemployed.
The underlying problem here is that it’s basically impossible to distinguish between someone naively allowing strangers to use their property and someone who is trying to cover their tracks by pretending to be unaware, and the internet is full of people who will abuse any free service. If it happens once, maybe you can plead ignorance but that’s not a foundation for a robust file sharing network.
I am not a mailman though, who surely have special protection. If you want an analogy, I would rather go the "carry a closed package for an anonymous stranger through airport security" danger way.
The danger is just too big, as my sibling commenter says.
If Uber Package driver delivers a package of drugs - pretty sure he won't get to jail as it will be trivial to prove he didn't put is there and had no idea.
In your mailman example, it would be like the FBI announcing exactly what the envelope looks like, uniquely identifiable, and if you encounter this letter to report it immediately or else your complicit. Or at least, that's the fear.
Yes, if it's encrypted you have no idea, but if anyone ever found out what was in it you can know who ever transmitted it. Unless everyone re-encrypts it repeatedly.. which then makes content address-ability awkward.
> Both of those could be enabled by combining data blocks semi-randomly. Ie. rather than downloading the data you're looking for, you download a block of data containing what you're looking for and some other data
What you're looking for is Oblivious Transfer or Mix Nets. Unfortunately, no practical solutions currently exist (except for Onion Routing).
> Remember that data can be encrypted by content hash, so there is no way to know what you're storing unless you go scouring the public internet for a key to decrypt it.
Censorship does not work this way. First, content that requires censorship resistance is usually public, otherwise one can employ a number of p2p or centralized solutions to share content directly.
Public content is easy to find, so step one: you find and identify the list of nodes that serve the content. Step two: you block the nodes on a level that's available to you (depends on the country). Step three: your users are unable to use the network effectively, because half of the nodes don't work, and revert to using Facebook.
It almost sounds like part of his concern is that the web server could view the content you are requesting. But this is always true for IPFS data since it's public and available to anyone with the CID.
Also you can use the CID (which is a hash), to verify the content hasn't been tampered with.
If I understand correctly, IPFS will even re-advertise blocks you have downloaded on the DHT. So per default your whole browsing history is public on the IPFS network.
> the gateway is in full control and can inspect and tamper with the data as much as it likes. And there is no way for the client to know or detect if it is happening.
Regarding data tampering;
IPFS uses content based addressing, the point is that the client _can_ know by computing a message digest of the incoming data. The verification is fast and cheap.
And yet the most widely used IPFS client, Brave, does not implement this.
The blog post is technically wrong, but I think it's worth accounting for the general state of the ecosystem in these decisions, and in whether to include IPFS in cURL, and it sounds like the general state of the ecosystem is not up to the standards that cURL should reasonably expect from a protocol asking for inclusion.
That's not what their documentation says. It says that it uses a public gateway, and prompts the user asking if they would like to use a local gateway, at which point it will download go-ipfs and run that. That's a fairly smooth UX to get a local gateway running, but I wouldn't bet money on it being used by the majority. Most people take the path of least resistance, and it's up to projects like Brave or cURL to ensure that the path is safe, which in this case it is not for Brave.
The hash you have is computed from the hash of the content and other data. You would need to send that additional data out-of-band to allow the client to compute the overall hash and verify it. (Not to mention with urls like ipfs://<hash>/some/path you only have the hash of some arbitrary parent node, so there's even more additional data necessary to be able to verify that the content at that path under the hash you have is valid).
You do not need to send the additional data out of band unless you are claiming that sha-2 (the hash function used in ipfs) is vulnerable to second preimage attacks.
It's not possible to verify a file downloaded from IPFS using only its CID, because an IPFS CID does not contain a checksum of the file content. It contains the checksum of a meta-file, which contains the hashes of further meta-files.
For example, debian-10.7.0-amd64-netinst.iso has SHA256 checksum b317d87b0a3d5b568f48a92dcabfc4bc51fe58d9f67ca13b013f1b8329d1306d. Here are two example CIDs generated from that file:
Not providing verification would seem like a nearly fatal flaw would it not? If one cannot pass around IPFS urls and ensure that the content returned is the same for all users, it would make a great way to MiTM a target.
As others have said, it's not as straightforward because the CID contains a hash of the data's encoding which is not the same as the data itself (files are chunked into pieces to facilitate incremental verifiability and better caching, the encoding supports directories, etc.).
pastel-mature-herring~> What are the issues listed in these comments about IPFS?
victorious-capuchin|> The main issues seem to be that IPFS is not yet up to the standards that cURL should reasonably expect from a protocol asking for inclusion. In particular, there are concerns about the lack of censorship resistance and user privacy.
The HERO will be when IPFS CLIs/Libraries have the hash verification included. CLI & Library open source devs, please be our heros. We want IPFS and security (both)
Great to see the proposed fix in ffmpeg. Using a remote gateway should really not be a default but something the user has to consciously and deliberately configure.
I will never run an IPFS gateway as long as there are no robust legal protections against someone provoking a downloads (and with IPFS, subsequent uploads) of content that's illegal to have on your hard drive (a universal example is CSAM).
And believe it or not - occasionally such content is used as a weapon. A Twitch streamer with an open Dropbox learned the hard way. It's used regularly on _that_ kind of imageboard to derail threads. I know because a naive version of me briefly ran an archive of a supposedly worksafe board about video games a decade ago and had their door knocked down at gunpoint about a year later.
I don't know how the Fediverse still exists, considering that most implementations are very eager to mirror all media they encounter.
I know HTTP can provoke the same caching, but for that there's actually much more robust legal precedent.
62 comments
[ 1.7 ms ] story [ 110 ms ] threadWith DoH generally must both opt in and must choose your provider.
Inspect, yes, but tamper? IPFS content IDs are hashes, so it seems like it should be possible to verify what you have been sent if you really cared enough to do so.
There isn't a tool today that can verify an IPFS address without actually downloading that file's DAG, which is basically downloading the entire file.
Imagine if I gave you a file out of a git repo, and the long commit hash. It's insufficient on its own to verify integrity, you need the intermediate nodes.
With ipfs you can infer that it used the reference chunker and try to reproduce the DAG but there is no guarantee it will work.
Better to just speak ipfs to the p2p net and get the DAG.
Git the Merkel root is a list of directories and hashs of whole files.
Ipfs, an individual file can be made up of many trees and those trees can have trees.
Example: https://explore.ipld.io/#/explore/QmbGtJg23skhvFmu9mJiePVByh...
That is a single file, but good luck verify that file without actually downloading the entire dag tree which includes the file.
Worse, that same file can be chucked and bucketed infinitely number of ways. So if I download that file and the re-add it to IPFS, I will very likely get a _different_ address. And that address will likely share _none_ of the of the nodes from the previous address. Meaning even though the same file is in IPFS, they are completely distinct and don't actually help seed each other.
And said proxy could bounce your request around for max 30 hops before a random IPFS-proxy honors the request. This is horrible for user privacy.
Brave browser [1] at least has a native support for IPFS and no need for proxies at all.
[1] https://brave.com/brave-integrates-ipfs/> By default, Brave will load the URI being requested via a public HTTP gateway; however, it will also show an infobar that asks you if you’d like to use a local node to resolve IPFS URIs.
So Brave by-default uses a public gateway.
It literally says: I have insisted, in the PR for ipfs support to curl, that the IPFS URL handling code should not automatically follow such gateway redirects, as I believe that adds even more risk to the user so if a user wants to allow this operation, it should be opt-in.
a better analogy than http urls would be git commit hashes.
The point of IPFS is that the server can't tamper with the content because it would break the hash, but if you're using an untrusted proxy then it is just as bad as using an untrusted proxy for your normal web browsing.
For the web, TLS provides tamper-resistance. Accessing the web through a HTTP proxy would allow tampering, similarly to accessing IPFS through a gateway.
It needs censorship-resistance added. It needs user privacy added.
Both of those could be enabled by combining data blocks semi-randomly. Ie. rather than downloading the data you're looking for, you download a block of data containing what you're looking for and some other data. Then nobody knows what it was you were looking at. Some combined blocks might contain a few bytes of data from thousands of files.
For censorship resistance, rather than users pinning individual files, the network needs to assign them blocks they must store (or be kicked off the network). Some of those blocks will be combined blocks which contain a few bytes some users may not wish to store - and that's fine, it means IPFS isn't for them.
Remember that data can be encrypted by content hash, so there is no way to know what you're storing unless you go scouring the public internet for a key to decrypt it.
http and tcp, other transport systems, also do not have censorship resistance. it's the wrong place for it.
you can use ipfs to build a censorship resistant system, just like you can with tcp, but that is the purview of a higher layer.
I don't think it's possible to reasonably to maintain layering while adding privacy and censorship resistance.
That IPFS does NOT put data I did not explicitly requested on my storage is exactly why I feel comfortable using it compared to some other distributed storage systems where I might end up with horrific data on my system.
And encrypted data you don't have the key for is indistinguishable from random noise. 'Combined' blocks that you don't know which partner blocks are necessary for decoding are likewise useless.
You're being like a mailman handling mail - you can't look into the mail, so you can't know if you're delivering love letters, junkmail, or terrorist propaganda.
ISPs and USPS have common carrier protection, by default a home user does not.
The underlying problem here is that it’s basically impossible to distinguish between someone naively allowing strangers to use their property and someone who is trying to cover their tracks by pretending to be unaware, and the internet is full of people who will abuse any free service. If it happens once, maybe you can plead ignorance but that’s not a foundation for a robust file sharing network.
The danger is just too big, as my sibling commenter says.
If Uber Package driver delivers a package of drugs - pretty sure he won't get to jail as it will be trivial to prove he didn't put is there and had no idea.
In your mailman example, it would be like the FBI announcing exactly what the envelope looks like, uniquely identifiable, and if you encounter this letter to report it immediately or else your complicit. Or at least, that's the fear.
Yes, if it's encrypted you have no idea, but if anyone ever found out what was in it you can know who ever transmitted it. Unless everyone re-encrypts it repeatedly.. which then makes content address-ability awkward.
This sentiment also applies to your argument, i.e. IPFS isn't fit for your purpose; and that's fine, it means IPFS isn't for you.
There are a whole bunch of censorship-resistant, plausibly-deniable systems out there (e.g. I run FreeNet from time to time)
What you're looking for is Oblivious Transfer or Mix Nets. Unfortunately, no practical solutions currently exist (except for Onion Routing).
> Remember that data can be encrypted by content hash, so there is no way to know what you're storing unless you go scouring the public internet for a key to decrypt it.
Censorship does not work this way. First, content that requires censorship resistance is usually public, otherwise one can employ a number of p2p or centralized solutions to share content directly.
Public content is easy to find, so step one: you find and identify the list of nodes that serve the content. Step two: you block the nodes on a level that's available to you (depends on the country). Step three: your users are unable to use the network effectively, because half of the nodes don't work, and revert to using Facebook.
Also you can use the CID (which is a hash), to verify the content hasn't been tampered with.
People, even in this thread, keep spouting privacy.. when i'm not sure if i've ever seen that as a goal.
If anything content addressability makes privacy remarkably difficult.
https://blog.cloudflare.com/e2e-integrity/
It looks like they've taken the FF addon down and archived the source repo though.
> the gateway is in full control and can inspect and tamper with the data as much as it likes. And there is no way for the client to know or detect if it is happening.
Regarding data tampering;
IPFS uses content based addressing, the point is that the client _can_ know by computing a message digest of the incoming data. The verification is fast and cheap.
And yet the most widely used IPFS client, Brave, does not implement this.
The blog post is technically wrong, but I think it's worth accounting for the general state of the ecosystem in these decisions, and in whether to include IPFS in cURL, and it sounds like the general state of the ecosystem is not up to the standards that cURL should reasonably expect from a protocol asking for inclusion.
You only have to verify the content yourself if it is not being verified by someone you trust (i.e. a local gateway that you're running yourself).
For example, debian-10.7.0-amd64-netinst.iso has SHA256 checksum b317d87b0a3d5b568f48a92dcabfc4bc51fe58d9f67ca13b013f1b8329d1306d. Here are two example CIDs generated from that file:
https://cid.ipfs.tech/#bafybeihjy54iyvheotna2aeqmzhqnro6yot4...
https://cid.ipfs.tech/#bafybeihfqpypuhmtyzazrj3g4b4f4nqk2ziy...
Notice that neither one contains the original checksum.
See https://github.com/multiformats/cid/issues/22
Here is IPFS's master issue for "verifiable gateway responses", which summarizes the technical challenges and progress: https://github.com/ipfs/in-web-browsers/issues/128.
victorious-capuchin|> The main issues seem to be that IPFS is not yet up to the standards that cURL should reasonably expect from a protocol asking for inclusion. In particular, there are concerns about the lack of censorship resistance and user privacy.
And believe it or not - occasionally such content is used as a weapon. A Twitch streamer with an open Dropbox learned the hard way. It's used regularly on _that_ kind of imageboard to derail threads. I know because a naive version of me briefly ran an archive of a supposedly worksafe board about video games a decade ago and had their door knocked down at gunpoint about a year later.
I don't know how the Fediverse still exists, considering that most implementations are very eager to mirror all media they encounter.
I know HTTP can provoke the same caching, but for that there's actually much more robust legal precedent.