Sure, it's easier to setup your own web presence with "the cloud" today than it used to be, but this only further centralizes control of content to the big cloud providers.
Not to mention the cost of bandwidth when serving content via HTTP. Sure, you can distribute your content via BitTorrent, but what kind of user experience is that? Can my grandma use it? Probably not.
While running an IPFS node does supply some utility to the DHT, the primary utility you can provide to the network is providing files.
You only provide files that you have previously accessed via IPFS or added yourself. So even without any external motivations to sponsor other people's content (which are in the works), you have the motivation of sponsorship of content that you wish to increase the robustness and accessibility of. Increasing access to public utility can be itself an incentive.
Reading this felt kind of like the first time I read a writeup on Bitcoin. There's the same sense of throwing out some old, formerly immutable rules here, the excitement of something that's going to test some boundaries and inevitably clash with some authority (how can you, for instance, comply with the EU's "right to be forgotten" when that information is scattered into a redundant mesh?). Interesting times ahead for IPFS.
Agree. What we really need is for browser vendors to get together, bring in the tremendous financial and technical resources they have, and get a decentralized protocol formalized and supported across browsers. It may take years, but it needs to be done.
OK, as apparently .js is the defacto thing to build cool sites, I for one will be left out here.
On sites not known to me, I typically block most (read all) .js at first, just to make sure that these sites do not leak too much information about me to external tracking tools, advertisers or the like.
So using .js seems to me a bit problematic in regards of people not enabling it by default (but this is an absolut minority, if I look at the stats of the sites I analyze).
Well, I think you can install something to use it without js, and they were just saying js would be for if someone wished to use it by simply using a webpage, with no install?
I use a custom hosts file (well, technically I run it as an internal DNS server) from Someone Who Cares to block most of the aforementioned annoyances: http://someonewhocares.org/hosts/
It works very well for me so I'd recommend it to others, if they're not already aware of it.
I've always wondered if Windows or other operative systems read the entire hosts file everytime they want to resolve an address. Maybe a big hosts file is bad for network performance?
It will largely be read out of memory, since it doesn't change much and is frequently requested. Scanning 64K or so won't break a sweat compared to doing a DNS lookup across the network to a resolver that might itself need to make a request.
The first implementation of IPFS is written in Go, and it includes fully functional nodes, http gateways, and libraries, but they're also working on a node.js version (that's also compatible with browser environments) so that non-technical users can have ipfs sites "just work". (although it requires at least static content served from a "normal" http/html server)
> OK, as apparently .js is the defacto thing to build cool sites, I for one will be left out here.
I doubt they're building it in JS for giggles.
In order to make changes like this stick my guess would be that you need to reach a tipping point before interest fades, and to do that you need to take away the browser vendors ability to say 'no' or 'maybe later'.
As you point out, people who don't run JS by default represent a tiny percentage of traffic, so would have to be a secondary concern to be supported later.
One solution for people like you (and me) who care about execution of too much js could be to make the code free software. You then can allow all free js in your browser by default. See [0] for details.
Interesting project addressing linkrot and central servers. Hopefully this line of work will attract more attention and developers.
Sadly the design contains some flaws that need addressing.
From their whitepaper (http://static.benet.ai/t/ipfs.pdf)
The architecture is essentially creating 1 giant Bittorrent swarm where each piece may be seeded by different people. See section 3.4.4 of whitepaper. This approach suffers from the "double coincidence of wants" problem. There will be a billion items if this takes off. Matches are rare and could take hours/days.
I haven't thought about IPFS in months. One thing I do remember thinking about months ago, are aggregators and distributed indexers.
I don't know enough about those kind of security attacks to comment.
My comment "you guys are great!" refers to their team as people not necessarily the project itself. I've worked with them before, and they are awesome folks.
Right to be forgotten however applies only to search engines right now, doesn't it? The articles the results used to point to still exist. So a redundant mesh itself doesn't seem to apply here.
The "right to be forgotten" applies to any information service, not just search engines. However, search engines are often easier legal targets since they do business in the EU and thus can be made subject to EU law. If I have no business in the EU, and post my website on a non-EU hosting provider, EU law can demand I take down a page but has no jurisdiction to force me or force my provider. But e.g. Google, since they do business in many EU member states (primarily sell advertising), if they disobey EU law, EU governments/courts can take legal action against them (e.g. confiscate their EU advertising revenue).
Furthermore the right to be forgotten only mandates hiding results when someone searches specifically for a name, for example. The results are still allowed to come up for unrelated search terms.
Directive 95/46/EC (the Data Protection Directive) never specifically mentions "search engines", it talks about "controllers" of personal data. The ECJ ruled (Case C‑131/12) that search engines such as Google count as controllers of personal data, and thus are subject to the "right to be forgotten" under the Directive. But clearly the Directive is applicable to many other things than search engines, so the "right to be forgotten" cannot be restricted to search engines either. (Newspapers should be exempt because article 9 of the directive has an exemption for "the processing of personal data carried out solely for journalistic purposes or the purpose of artistic or literary expression", but search engines don't fall into that category, and other web sites may or may not either.)
The court judgement - http://curia.europa.eu/juris/document/document_print.jsf?doc... - paragraph 26 says "As regards in particular the internet, the Court has already had occasion to state that the operation of loading personal data on an internet page must be considered to be such ‘processing’ within the meaning of Article 2(b) of Directive 95/46" - so by the Court's reasoning the Directive potentially applies to any web page, not just search engines.
Indeed, the Data protection directive is not about search engines: it is about privacy online in general and any web page published or plausibly controlled from within the European union is subject to it (and of course also to any IPFS page, although who controls it is harder to say). It has always applied to a lot more than search engines.
The "right to be forgotten" is a phrase used in the argument for de-listing, and later became the name for the ruling itself. This was however not a right that the court granted, but rather the court points out that the data protection directive stipulates that personal data shall be:
"""
(c) adequate, relevant and not excessive in relation to the purposes for which they are collected and/or further processed;
(d) accurate and, where necessary, kept up to date; every reasonable step must be taken to ensure that data which are inaccurate or incomplete, having regard to the purposes for which they were collected or for which they are further processed, are erased or rectified;
"""
and judges (in paragraph 93) that in particular, data that was once in compliance might not be at a later time.
It also states that the failure of the particular data in this case to be those things means the person described by them can invoke the right to object to the processing, in this case having personal data listed on a page as a response to a "search made on the basis of his name" (in paragraph 94).
It is not completely clear, but I think the query being judged upon includes only the name of the person, and adding a keyword relating to the particular year, or bankruptcy actions, would be enough to make the data "adequate, relevant and not excessive".
More relevant, is that it seems safe to include the link even in name searches that also includes additional keywords, since google lost on account of them judging a very old news about him still relevant for listing in a general name query, and the court saying: not so.
Isn't the right to be forgotten something, that esp. relates to search engines? They (esp. Google) have to delete the links to this content, but the content can stay on the web forever.
So this isn't per se a problem. But what happens, when I act as a "mirror" in this net and have mirrored some questionable content (who ever might define questionable is up to anybody to imagine). Will I get a shake down from the authorities, as oftentimes happened with people having tor exit nodes?
I fear that the clash would come rather sooner then later, as a lot of people do have a lot of incentives for a centralized web (money, politics, control, power).
But problem is that it is. It's a EU law, so while I agree with you, I sense the IPFS will clash with this law. Let's wait and see, progress can't be stopped though...
But problem is that it is. It's a EU law, so while I agree with you, I sense the IPFS will clash with this law. Let's wait and see, progress can't be stopped though...
> It should be shaped by what the majority think is right for society.
I would kindly suggest that we consider examples from history when laws backed by the majority have resulted in horrifying, depraved violence and deprivation; then contemplate whether it is indeed moral to shape law through majority opinion.
As far as I understand it, you must issue a "pin" command in order to relay or "host" contents. So if some contents is illegal, authorities can command you to "unpin" it. It's a better option than having to take down your whole node.
>for instance, comply with the EU's "right to be forgotten"
Easy: The Net interprets censorship as damage and routes around it.
Oh, right, we're not calling this censorship yet. Funny how quick we are to put up mirrors to Tibet and Tank Man content but somehow we're on the hook to protect London businesspeople and corporations from criticism? Like Bitcoin or Bittorrent and other decentralized applications, it will not be able to follow the dictates of nation states and it will be on nation states themselves to filter appropriately. There's no one to call to remove "unwanted" content. No one to hit with fines, taxes, lawsuits, and fees.
Also, there's a discussion to have in regards to EU shakedowns of US companies. These billions of dollars in fees that go straight to the EU don't seem like consumer protection to me, but fundraising. Billions of dollars in fees for "bundling media players" or "having an Android monopoly" should give any wannabe entrepreneurs pause. Its a shame that the EU is unquestionable on sites like HN and reddit. There's a lot of questionable morality here and obvious signs of corruption. So yeah, not being able to be dictated by the EU or any nation state is a feature, not a bug.
How dynamic? Resources can be updated to point to new immutable objects. For instance, as an operator of a website, I can sign the resource `/comments/10189265.json` as referencing an object with a particular hash, and then when there are additional comments, change the object it references. How fast things like this can update are dependent on the implementation, which I haven't read.
More work would be needed here perhaps. But decentralized apps should happen one day. As someone without real technical knowledge in security and decentralized platforms, I'd love to hear from some on HN where we really stand on decentralized apps and security.
- Public data. Distributing, searching and querying is not a problem.
- Restricted data. You'd be distributing encrypted data to peers. Now, if we have a way to send an encrypted query to the peer and have it run on encrypted data without the peer being able to understand the query or the data, we'd be set perhaps.
- Obviously this would need application architectures to change. But that's solvable.
> Now, if we have a way to send an encrypted query to the peer and have it run on encrypted data without the peer being able to understand the query or the data, we'd be set perhaps.
Is that possible? It doesn't seem to be "N versus NP" but it seems similar. How can a computer find a correct answer to a question it doesn't know?
I suppose you could send off small units of processing across many nodes, such that none of them knows the full result set...
Since the private data and the query are from the same user, they could share some mathematical connection perhaps, which would allow a query to be done and the results sent back (results which only the user can decrypt).
Google shows up plenty of articles on the subject, but I am not an expert to know where we stand on this.
Interesting. So you'd perform operations on encrypted data, and send back an encrypted result that's useful yet different from the original encrypted data, all without ever performing decryption?
My understanding is that it is possible, and Turing-complete systems can be built, but they are slow and unwieldy, requiring a large amount of cyphertext to encode data (on the order of 4KB per float, I seem to recall).
It's still up for debate whether or not such techniques will eventually be up for general distributed computing, e.g. distributing a VM instance over multiple distributed servers; or just for specific use-cases, e.g. bitcoin-esque distribution of small transactions, etc.
This is called "homomorphic encryption". There's a lot of research in this area, but we're far from being able to efficiently perform all computations.
IPFS is designed with mutability in mind-- think of it like git branches which advance to newer versions. (though you can indeed get rid of version history in that pointer if you want). So, the short answer is "yes, IPFS will work with mutable content just fine", though _how_ the mutability works is a harder, deeper question.
I think the best sources will be the paper, the talks on the website, and the FAQ:
With IPNS, which allows the creation of links whose destination can be mutated by whoever has the private key, you can implement quite a lot of interactivity. You won't ever be able to implement facebook over this platform, but you can replicate all of its functionality by using a tumblog-like architecture. Facebook's design is inherently centralized, and therefore doesn't really fit into the world of IPFS.
The idea is that every user owns their own content, not some central organization. Every user maintains a log of all of their actions, like an RSS feed. Every time they make a status update, that's an item. When they comment on someone else's status update, that's an item. Even when they just like something, it's an item. Each update of the action log would also have send ping to all friends, over some kind of distributed IM like tox. They already need an IM connection like that for messenger.
Given that information alone, anyone can reconstruct the activity of all their friends into a news feed, and post new content. You'd need a lot of client-side leverage, and the ability to upload to the IPFS network from the client, but all the dynamic information is there.
And there you have a fully distributed facebook clone. More resilient and privacy-sensitive and monopoly-averse than even tools like diaspora and statusnet.
Isn't friend recommendation mostly stuff like "people who follow you/you follow also follow X"? Maybe going one or two more follow hops, but not so far out that you need a global view of the social graph. If users of a decentralized network publish their following and follower lists, it should be pretty easy for a local crawler to check those hops.
You have friends. A script can check your friends, and make suggestions based on that.
You won't have the full graph that Facebook does, but does that really matter? I suspect the best predictors of who your friends are would be people already in your network.
Not really. There's no reason you couldn't implement Facebook's core functionality with personal servers containing a profile at each node and a pub/sub (or even just pull) system connecting each node with its friends.
Yes, I believe diaspora was a social network of this sort. But it's just one example, and not an indictment of decentralized social networking in general. Your address book + SMTP is a decentralized social network that caught on just fine. The only difference is that it was designed and built before corporate interests were the driving forces behind new standards (or lack thereof).
Actually, this is exactly what I'm looking for. I want to be able to control my photos and other content, and share that with family and friends.
All data is encrypted and my server can be hosted anywhere (just as simple as setting up a WordPress instance).
My own "news feed" are a combination of all of the other nodes I'm subscribed to.
My problem with Disapora is that they push the "sign up to someone else's pod" rather than "install your personal pod".
It just feels like a multi-centralised model. Rather than one Facebook with my data, now there are 311 "Facebookesque" Diaspora Pods (as of today).
As much as people hate the idea, a WordPress like (PHP) application as a "personal server" would be successful. There are lots of hosting companies out there where you can get up and running for peanuts.
My aim would be for a simple install:
- Download software and upload to hosting company
- Enter MySQL auth details
- Choose local filesystem or third party (S3, DropBox, Google Drive, MS SkyDrive) with OAuth as the backing store (files stored encrypted).
- Finish
Then I would be able to subscribe to other friends and family personal servers, just like RSS feeds. Friends can access my own personal server through OAuth and leave comments on content. Those comments are echoed back in my private feed which is displayed in their merged news feed on their own app instance.
The technology is already there. It just needs someone who is damn good with PHP!
We're heading towards this with Peergos (https://peergos.org source at https://github.com/ianopolous/Peergos). Encrypted, self-hostable file storage, sharing and social network. We're considering using IPFS under the hood for fragment storage/finding. You'll be able to either use your own disk, or enter credentials for a cloud provider. It's still very early days yet though.
The reason why they get people to sign up at other people's pods is that it's easier to get users that way.
They are struggling to get users regardless and that is their biggest flaw.
I'm thinking about the discovery problem and how this relates to distributed technology now. File storage looks solved but creating content relationships like lists of similar files sounds much harder and still reliant on a authority that curates and filters the spam. Lost data in IPFS will be "buried treasure" in that nobody has the link to it anymore, but it still exists in a literal sense.
- Is it possible to, say, log into my bank account, see my current balance, and set up bill pay?
- Can I order a pizza over IPNS?
- Can I do realtime chat?
Also, this article says we could use Namecoin and not rely on ICANN for DNS, however, you still need ICANN to distribute ip address to get connected to the internet in the first place. Or am I missing something?
I think that it should be possible to build hybrid centralized/decentralized sites with IPFS, with static public content being fully decentralized and sensitive information going through direct tunnels. Of course, the more decentralized your system the better.
As far as the IP system, yes we are unfortunately still reliant on ICANN being sensible about IP allocations, but the DNS system can be fully deprecated. I do know that there is a project out there to redo the IP system in a decentralized fashion but I can't recall its name right now.
The first two cases you mention are about communicating with a specific real world organization. You can implement it using ipns, but it's inherently centralized.
The last one is a no. Well actually yes if you just poll a file, but that's not practical on a large scale. You need an external communication channel to implement that. It's possible to have that be fully distributed as well though, as in tox. You need this external communication channel to implement a lot of things anyway, so it would probably be part of the platform that forms around the idea of a distributed web.
I'm very excited to see what happens with IPFS. The article talks about replacing HTTP however, and this is definitely a tricky task.
Someone in this thread already asked one of my questions (So is this primarily for static websites?) but my second question is: So is this primarily for personal websites?
I'm having a hard time finding a good way for Facebook, for example, to monetize their website. Targeting ads go out the window with mostly static content. Even more so though, what about Netflix? How is DRM done? How do you make sure only the correct users can access your objects?
edit: Also, doesn't a "permanent web" have an inevitably fatal flaw that you can't free space?
One monetization strategy in general for distributed networks is that users pay for bandwidth. They're rewarded for the bandwidth they provide. They profit if they provide more bandwidth than they use. And app developers are rewarded when their apps cause bandwidth to be used. I don't know if IPFS works this way.
As far as paying a premium over raw bandwidth to access premium content (Netflix), you could pay them to unencrypt it for you to download. DRM though isn't secure in the first place and decentralization won't make it better.
Oh, of course, I'm not trying to argue that DRM is good or that decentralization would make it better or even work as well. The problem of course is that without it working as well as it currently does, premium content providers have no incentive (and in fact, are decentivized) from making the switch.
The general idea of decrypting premium content at a cost would work fairly well, however I'm wondering if it's something that can be done easily and dynamically via IPFS. For example, can I, without burdening the network with a billion copies, encrypted with each the client's key, hand that data off to them? If I simply decrypt it and hand it over, it's on a distributed network and anyone can grab it.
Delivery of the unencrypted premium content would likely happen through a direct link with a different protocol. As you say, the content providers don't want that content to be accessible to everyone in the network.
The website used to navigate the catalog of premium content could be built as a IPFS page though.
You can free space, but how that happens is not explicit.
The more people a file is requested by, the more places it is stored. If the file ceases to be requested, and all the participants decide to remove it from their caches, the file is gone.
DHT records are slightly more persistent, but without a sponsor they fade as the network's churn removes the nodes that once hosted them.
It is very hard to delete things, and the most likely things to be deleted are things people are not providing effort to preserve (even one person providing such effort is sufficient to maintain the DHT records.)
Hmm, that was sort of what I figured, but that seems to me as though calling it a "permanent" web is a bit of a misnomer. There is still a place for projects like the Internet Archive.
Interesting point - but why wouldn't the business model that worked for Facebook - ads - work on a decentralised web? Instead of posting content to FB alongside which they run ads and generate revenue, individual publishers could keep the ad revenue themselves, perhaps with a brokerage / commission model like Google operate. Seems to work OK for blogs, which are quite decentralised. Why shouldn't it work for status feeds?
In terms of targeting, once you're dealing with "small data" instead of centralised "big data" you need a lot less algorithms to figure out what ads might work. A FB neural net probably does a lot of deep learning to guess that a bunch of mountain-biking friends might be interested in saddles. Decentralised operators could probably figure that out themselves.
My first thought when reading the specification was that they're liars and making bold claims out of context. Yes, any torrent ever released is also permanent. Yet, if I dig my archives and try to download 10 years old ed2k links from ShareReactor, somehow there aren't any seeds. Wonder why? Isn't it permanent? Because the ed2k link is right for sure, it's permament.
It will reduce the probability of a collision at the expense of the address being longer - because now you need to carry around hash1 + hash2. But still it doesn't eliminate the possibility, only makes it less likely.
Exactly, what I noted was the most likely temporary/possibly long-term solution to the problem.
What about hashing an object that contains the hashes generated by x hashing algorithms? an extra step of resolution, sure, but some way to easily demarcate that some file experienced a collision with another file, and had to have the number of hashes increased to stave off further collisions?
There's no hashing construction that will "completely eliminate the possibility" (see https://en.wikipedia.org/wiki/Pigeonhole_principle), so I'm not sure what answer you're looking for. It's computationally infeasible to generate collisions (accidental or malicious) if SHA-2 is not broken. If you're interested in reading about how combining multiple hashes affects security, see this paper: https://www.iacr.org/cryptodb/archive/2004/CRYPTO/1472/1472....
We're way past my personal knowledge, but what I was thinking was that the probablistic combinations of two algorithms outweigh (in terms of probability of a collision) the likely hood of a single algo.
IE if you have some hashing function with a 1/2^5 chance of a collisions, using that in conjuntion with another hashing function with say 1/2^3 requires that both unlikely probabilities occur, resulting in 1/2^8, where a "stronger" hashing function might only have 1/2^6 (technically stronger than either one of them, but still not as strong as both)
I'd really love some correction on the logic above though, I am by no means well studied in probability or hashing functions, despite some effort, and would really appreciate any corrections.
All you're doing then is fiddling with probabilities. You can do that just as well by changing the length of the sha unless you have a reason to think sha is broken.
However, the probability of a hash collision is already incredibly low. As a demonstration of this, here's a list of a tiny subset of bitcoin private keys: http://directory.io/
The human mind just isn't capable of understanding how large of a number the number of sha hashes is.
Right, but fiddling with probabilities is exactly what you want to do in that case right? I don't know if a hashing algorithm that has no collisions exists (I suspect the answer is no, maybe other than the identity function). If collisions start to become a problem, you can move that problem further out (much much farther out) by mixing more hashing functions in, which was my point.
It's well documented (of course also by the link you posted), the super low likelihood of hash collisions, but I think (hope) the original poster knew that -- was trying to answer assuming that extraordinary case actually happened.
> I don't know if a hashing algorithm that has no collisions exists (I suspect the answer is no, maybe other than the identity function).
There is no hash function without collisions. The set of inputs is infinite but the set of outputs is finite. (The identity function isn't a hash function--its output isn't a fixed length.)
ipfs currently uses what we call a 'multihash' where every hash value is tagged with the hash function used to generate it. As of now, the default hash function is sha256, and if I recall correctly, the chances of a world ending asteroid hitting earth and wiping out the planet is far more likely to occur than a collision. If sha256 is shown to be broken, we can change our default hash function to something like sha512 without breaking any existing data.
Well.... while an asteroid annihilating life on the planet isn't likely, the size of the web isn't a joke either, so such a collision is more probable than in other cases...
I suppose this could be made a problem for the user, i.e. if you want to make sure that your page isn't colliding with any other, then simply check first. (Of course there is still a race condition possibility of two colliding addresses being created at around the same time...)
Its generally considered safe to ignore, there are no known sha256 collisions, and most people are fairly confident they wont see one for quite some time.
the caveat of course being that SHA256 may be _broken_. That's when we'd have to upgrade to another hash function.
(We may improve how this upgrade will work down the road by allowing links to link with multiple hashes at the same time, but at this time this has not proved to be necessary, and can be added later)
256 bit hash collisions have so low probability that you don't need to prepare for them. If one happens, its cause for celebration.
You can cover earth surface with 8.5×10^36 oxygen atoms. Imagine that every one of them is IPFS file with 256 bit hash. Probability of collision is still less than 0.1%.
Well, I'm not 100% convinced this is going to take off, but I'm intrigued. I'm going to make an effort to get Fogbeam Labs' website up and running on IPFS shortly as well. I'm curious to see where this goes.
This is great news. Dapps [1] are coming, and DHT [2] /storage technologies like IPFS (although IPFS is more than just a DHT) are the other much-needed side of the coin - no pun intended - to make that reality happen. Exciting times ahead.
[1] decentralized app
[2] distributed hash table
That's interesting, but I see no reference in the website or the paper to GNUnet or Freenet or other existing DHT-based file distribution networks. How does this compare?
I was running IPFS as "my own pastebin for files" for a while (it's great!), but was wondering what can they do to improve adoption / popularity. This move is amazing! Useful, interesting for people who care, and visible for others.
I'm assuming that once ipns starts to work reliably you'll add the support for it? Or is there something that would prevent it? (I mean for single-user repo anyway - multi-user would have to share the key... somehow)
I have always wondered if there is some way to make a distributed network which provides content in such a way that users' freedoms are maximized. I don't particularly like handing over all control of my data and internet presence to cloud providers. IPFS looks really, really cool!
As ot asked, I am also curious about how this compares to other efforts to create decentralized networks, like Freenet and GNUnet. I definitely plan to pick one in the coming weeks and start using it, hope this catches on.
I wish this article mentioned something about BitSwap; before reading the IPFS paper, I was doubtful about anyone wanting to participate without incentives.
Still, even after skimming the paper, it sounds like a lot really depends on how well BitSwap is implemented. At one point, the paper says this:
> While the notion of a barter system implies a virtual currency could be created, this would require a global ledger to track ownership and transfer of the currency. This can be implemented as a BitSwap Strategy, and will be explored in a future paper.
So it kind of sounds like they need something like bitcoin, but don't want it to be tightly coupled to bitcoin - which is probably smart. But without a single clear solution for a reliable BitSwap strategy (and I'm not sure how it could be reliable without a distributed global ledger of some kind), it's hard to see how/why the necessary resources will be contributed to the IPFS network.
I absolutely love the concept and the underlying moral reasoning for why we need IPFS!
One of the first questions that came to me: Is there any way to add a forward error correcting code, like an erasure code to IPFS? I didn't find any discussion of this in the IPFS paper.
This seems somewhat critical to be able to compete with modern centralized storage systems which are likely to use FEC extensively to provide the level of redundancy that customers expect. Modern FEC codes can provide phenomenal levels of redundancy with less than 50% overhead. IPFS seems to rely on many fully-redundant copies?
Interesting Idea! I think TAHOE-LAFS also uses something like this?
From a practical standpoint, I'd like my local node to have a full copy so that my friends coming over don't need to reach out to the (maybe offline) internet but I don't see why we shouldn't support encoding the chunked blocks in such a way?
Based on my reading of this article, IPFS doesn't specify how you store the content - just that the content you provide back matches the requested hash. So in theory this could easily be implemented under the hood.
Though perhaps I'm misunderstanding your question - if you want multiple IPFS nodes to coordinated on storing parts of an erasure encoded file? that would be useful too. Again I think it's possible - you could build a small distributed system to host files this way. The most interesting variant though is if IPFS could have support for something like this natively - so nodes run by different users can each pitch in a little. Conceptually it's possible, but probably the protocol doesn't support it.
So let's say I'm building a decentralized chat app built on some DHT and would like to host it on IPFS.
I imagine getting the static html/js/css for the client-side app onto IPFS should be simple enough, but how would I handle bootstrapping?
Even in a decentralized service, clients need to connect to a seed server somewhere to start querying for other peers on the DHT. Would it be possible to host such a seed server on IPFS? Or would the seed server still need to live on a traditional server somewhere on the "old web"?
The way our network currently bootstraps is to dial a set of peers that are hardcoded into the default config file. We also have mdns for finding peers that are on your LAN without the need for any hard coded addresses.
Thanks for the quick reply. I was actually thinking of an app running on a separate IP-based DHT, but your reply brought to my attention that IPFS actually already runs on a DHT.
Since IPFS runs on its own DHT, are there any plans for a client-side IPFS DHT library for apps to use to connect to peers on the IPFS network? Similar to https://github.com/feross/bittorrent-dht, but with native support for IPFS peers?
Something like this would make building a client-side real-time app based on IPFS a much more streamlined experience since developers won't have to depend on an external DHT.
We've been thinking about this problem for quite some time. My friends and I are passionate about re-decentralizing the web. The biggest challenge we've come across is this:
Many users want to do things with other users
That means there has to be a way to persist state and history (of a group activity, an evolving resource, etc.) across machines. Mental poker is still very hard to do for groups. Since most clients will not have amazing uptime, that means you still need a network of servers. Servers that manage security behind a firewall.
So, the conclusion is -- for group writable data, the best you can do is have servers be completely interchangeable, but you will still need servers and networks of servers.
The good news is, many of the challenges are orthogonal to each other. Someone's building a Mesh network, for instance. Someone else is building the protocol layer with IPFS. We're building the social layer for this coming web.
Couldn't you do group editing through using lisp inspired link lists? See a comment thread? CONS your comment onto the end of the comment thread and push into the ether, and notify about the new list head.
Take other patterns from clojure immutable data structures for various other kinds of group edited time travelling sites
It might be possible to do some sort of self contained data structure that authenticates everybody and only allows authorized access to it, which solves the Byzantine Generals problem like BitCoin does, and which resists forking and instead uses vector clocks and Operational Transformations to consolidate actions done to the tree. But even so, how would it resolve conflicts? It could be a long time until a conflict is discovered, and then do edits get dropped, or what?
Why would you want to resist forking, and incur all that other complexity when you can just have an "authoritative" source for the entry point? Similarly, if someone wants to fork the conversation off and become a new "authoritative source" why would you need to stop them?
as for conflict resolution, merging, etc, it kind of looks like a (mostly) solved problem. I don't see how conflicts could occur in a comments stream. It's append-only.
It's append only until B and C get a copy and each append their own comments, then D and E get two separate forks and append their own comments. That can work for trees to which you only append, such as comments. That does NOT work for any operations that do non-append operations on trees, such as operations which rearrange the tree into a line.
So yes, for append-only tree structures, this may work. In terms, of say, a Google Doc that means you can never delete or insert any text, but only append text to the end of paragraphs etc. Certainly very limiting.
Unless there's something you haven't described, in which case I encourage you to provide a more in-depth comment describing a solution for actually inserting or modifying the tree.
actually, google docs ARE append only. They use the vaunted google wave tech of Operational Transforms. every single timestamped keystroke is saved separately in the database. when two users are involved, and editing simultaneously, the OT algorithms resolve the merge conflict on the spot. it kind of looks like a tree structure, or an append only array. the actual document you are looking it is actually something like a flattened keyframe in a movie that's reached the end.
In my view Maelstrom instantly disqualifies itself because it's kept closed source. I really don't understand how BitTorrent (the company) can take itself seriously preaching about "truely neutral" networks.
I've worked with IPFS, and it's really amazing. The development on it is very active, and the main developers are easily accessible for questions on Freenode#ipfs. I've been trying to build a tiny consensus protocol on top of it, but its native support for signing is still a little rough around the edges (and of course I've been distracted by a thousand other side-projects).
Maelstrom was a massive disappointment in my opinion, you appear to be locked in to static websites with no mechanism to issue updates. The entire site is packaged up into a single torrent blob meaning there is no hope of sharing swarms between different sites. And as has been said, it's closed source and Windows only. Avoid.
- So I want to download a file over IPFS today, how do I go about it? What sort of client do I use?
- Now I want to view a whole web page. Is that possible?
- Let's say I have a site that I want to publish through this, along with a bunch of files (images, downloads). I'm serving that site now through Apache. Do I need to 're-publish' each file (using the command in the article) every time a file on there changes? Or is there some automated way to have a distributed version of the latest version of my site? I mean, it's fine and dandy that people can store cached copies, but what I want is that if there's an update to my software, that people actually get the latest version when they click 'download', not that they get an old version because some dude somewhere made a mistake at one point 'hosting' a static version of my site and not updating, and visitors just happening to use that version because that dude is closer to them (in network topology terms) than I am.
You can use it like a unix command line tool to add and get files but you can also run it in daemon mode. It then also opens a HTTP server locally which allows you to use the same URLs as above just with http://localhost:8080 as the schema and host.
2) You can use the ipfs cli tool to add a directory with your sites html and other static content. The client will then give the the root content hash. Then you open /ipfs/$rootHash/index.html or /site2.html and voila! :)
Btw, this article is serving from ipfs. Try 'ipfs get QmNhFJjGcMPqpuYfxL62VVB9528NXqDNMFXiqN5bgFYiZ1' and take a look in that directory on your harddrive. Also: when you now run 'ipfs daemon' your node will help with requests for this content.
3) To update content with a static name, you can use IPNS, it's also mentioned in the article. IIRC: To resolve those entries (and get the latest ipfs content hash) the node with the corresponding key for the ipns entry needs to be online. If it isn't, you get an error like 'Path Resolve error: could not resolve name.'. Maybe come to the #ipfs irc channel on freenode for more specific questions.
Just reading back though the specification of IPFS, be wary about the claims of 'permanent Web'.
Distributed content in IPFS is eventually purged from nodes unless one of the hosting nodes has pinned it to be retained. Therefore, if no-one at Time x views certain content and pins it, then unfortunately at Time x + n that content might disappear just as thoroughly as under HTTP.
Unfortunately I fear that means that 'popular' content persists whilst niche and unique data might still fade away.
If somebody put something online fifteen years ago, they might not care about it anymore, but I might. With the current web, I can mirror that content by downloading it and putting it on a new server. With this system, that process would be part of the basic workings of the system.
With bittorrent you could make a call in a public forum for more seeds for a file. And if people have a copy of that particular content they can reseed.
I wonder if this would also be possible in this case?
For example, my small server might serve 200 sites but I'll backup around 500 or so older ones somewhere else.
248 comments
[ 3.3 ms ] story [ 239 ms ] threadSure, it's easier to setup your own web presence with "the cloud" today than it used to be, but this only further centralizes control of content to the big cloud providers.
Not to mention the cost of bandwidth when serving content via HTTP. Sure, you can distribute your content via BitTorrent, but what kind of user experience is that? Can my grandma use it? Probably not.
I hope IPFS sees further adoption.
and, you'll love what we have coming. Soon, you won't need to install ipfs at all :D
It's coming along, not fully functional yet but you can watch its progress (or contribute code :D ) at https://github.com/ipfs/node-ipfs
You only provide files that you have previously accessed via IPFS or added yourself. So even without any external motivations to sponsor other people's content (which are in the works), you have the motivation of sponsorship of content that you wish to increase the robustness and accessibility of. Increasing access to public utility can be itself an incentive.
That would change the internet, and the world.
- very soon, you wont need to install anything to use IPFS. it will "just work" with js on today's browsers.
- for best perf, yes, we need browser implementations. and... those have begun :)
Do you have links to browser implementations?
On sites not known to me, I typically block most (read all) .js at first, just to make sure that these sites do not leak too much information about me to external tracking tools, advertisers or the like.
So using .js seems to me a bit problematic in regards of people not enabling it by default (but this is an absolut minority, if I look at the stats of the sites I analyze).
It works very well for me so I'd recommend it to others, if they're not already aware of it.
I doubt they're building it in JS for giggles.
In order to make changes like this stick my guess would be that you need to reach a tipping point before interest fades, and to do that you need to take away the browser vendors ability to say 'no' or 'maybe later'.
As you point out, people who don't run JS by default represent a tiny percentage of traffic, so would have to be a secondary concern to be supported later.
[0] https://www.fsf.org/campaigns/freejs
Sadly the design contains some flaws that need addressing. From their whitepaper (http://static.benet.ai/t/ipfs.pdf) The architecture is essentially creating 1 giant Bittorrent swarm where each piece may be seeded by different people. See section 3.4.4 of whitepaper. This approach suffers from the "double coincidence of wants" problem. There will be a billion items if this takes off. Matches are rare and could take hours/days.
"Aside from BitSwap, which is a novel protocol". These types of protocols have been worked on for over 8 years. Nobody has ever been able to protect them from an Eclipse or Sybil attack. Early work: http://www.seas.upenn.edu/~cse400/CSE400_2005_2006/Wang/Writ... Deployed system from my research group, improved upon for 7 years now: http://www.pds.twi.tudelft.nl/~pouwelse/A_network_science_pe...
I don't know enough about those kind of security attacks to comment.
My comment "you guys are great!" refers to their team as people not necessarily the project itself. I've worked with them before, and they are awesome folks.
Furthermore the right to be forgotten only mandates hiding results when someone searches specifically for a name, for example. The results are still allowed to come up for unrelated search terms.
The court judgement - http://curia.europa.eu/juris/document/document_print.jsf?doc... - paragraph 26 says "As regards in particular the internet, the Court has already had occasion to state that the operation of loading personal data on an internet page must be considered to be such ‘processing’ within the meaning of Article 2(b) of Directive 95/46" - so by the Court's reasoning the Directive potentially applies to any web page, not just search engines.
The "right to be forgotten" is a phrase used in the argument for de-listing, and later became the name for the ruling itself. This was however not a right that the court granted, but rather the court points out that the data protection directive stipulates that personal data shall be: """ (c) adequate, relevant and not excessive in relation to the purposes for which they are collected and/or further processed;
(d) accurate and, where necessary, kept up to date; every reasonable step must be taken to ensure that data which are inaccurate or incomplete, having regard to the purposes for which they were collected or for which they are further processed, are erased or rectified; """ and judges (in paragraph 93) that in particular, data that was once in compliance might not be at a later time.
It also states that the failure of the particular data in this case to be those things means the person described by them can invoke the right to object to the processing, in this case having personal data listed on a page as a response to a "search made on the basis of his name" (in paragraph 94).
It is not completely clear, but I think the query being judged upon includes only the name of the person, and adding a keyword relating to the particular year, or bankruptcy actions, would be enough to make the data "adequate, relevant and not excessive".
So this isn't per se a problem. But what happens, when I act as a "mirror" in this net and have mirrored some questionable content (who ever might define questionable is up to anybody to imagine). Will I get a shake down from the authorities, as oftentimes happened with people having tor exit nodes?
I fear that the clash would come rather sooner then later, as a lot of people do have a lot of incentives for a centralized web (money, politics, control, power).
But the right to be forgotten is a flawed concept to me, and I think any tech that makes its implementation more problematic will do us a favour.
I would kindly suggest that we consider examples from history when laws backed by the majority have resulted in horrifying, depraved violence and deprivation; then contemplate whether it is indeed moral to shape law through majority opinion.
Easy: The Net interprets censorship as damage and routes around it.
Oh, right, we're not calling this censorship yet. Funny how quick we are to put up mirrors to Tibet and Tank Man content but somehow we're on the hook to protect London businesspeople and corporations from criticism? Like Bitcoin or Bittorrent and other decentralized applications, it will not be able to follow the dictates of nation states and it will be on nation states themselves to filter appropriately. There's no one to call to remove "unwanted" content. No one to hit with fines, taxes, lawsuits, and fees.
Also, there's a discussion to have in regards to EU shakedowns of US companies. These billions of dollars in fees that go straight to the EU don't seem like consumer protection to me, but fundraising. Billions of dollars in fees for "bundling media players" or "having an Android monopoly" should give any wannabe entrepreneurs pause. Its a shame that the EU is unquestionable on sites like HN and reddit. There's a lot of questionable morality here and obvious signs of corruption. So yeah, not being able to be dictated by the EU or any nation state is a feature, not a bug.
Edit:
If someone like GitHub supported this for GitHub pages it would be a great step forward for this as well.
I imagine you could go through their codebase to try to see how far this has come along: https://github.com/ipfs/go-ipfs
According to this, it looks like the implementation is currently in progress: https://github.com/ipfs/go-ipfs/blob/master/dev.md
Hope something in there helps you.
More work would be needed here perhaps. But decentralized apps should happen one day. As someone without real technical knowledge in security and decentralized platforms, I'd love to hear from some on HN where we really stand on decentralized apps and security.
- Public data. Distributing, searching and querying is not a problem.
- Restricted data. You'd be distributing encrypted data to peers. Now, if we have a way to send an encrypted query to the peer and have it run on encrypted data without the peer being able to understand the query or the data, we'd be set perhaps.
- Obviously this would need application architectures to change. But that's solvable.
Is that possible? It doesn't seem to be "N versus NP" but it seems similar. How can a computer find a correct answer to a question it doesn't know?
I suppose you could send off small units of processing across many nodes, such that none of them knows the full result set...
Google shows up plenty of articles on the subject, but I am not an expert to know where we stand on this.
It seems possible but extremely tricky.
My understanding is that it is possible, and Turing-complete systems can be built, but they are slow and unwieldy, requiring a large amount of cyphertext to encode data (on the order of 4KB per float, I seem to recall).
It's still up for debate whether or not such techniques will eventually be up for general distributed computing, e.g. distributing a VM instance over multiple distributed servers; or just for specific use-cases, e.g. bitcoin-esque distribution of small transactions, etc.
I think the best sources will be the paper, the talks on the website, and the FAQ:
- https://ipfs.io/ipfs/QmR7GSQM93Cx5eAg6a6yRzNde1FQv7uL6X1o4k7... - https://ipfs.io/ - https://github.com/ipfs/faq/issues
Freenet has a similar immutable datastore but chat, microblogging, etc have been built on it.
The idea is that every user owns their own content, not some central organization. Every user maintains a log of all of their actions, like an RSS feed. Every time they make a status update, that's an item. When they comment on someone else's status update, that's an item. Even when they just like something, it's an item. Each update of the action log would also have send ping to all friends, over some kind of distributed IM like tox. They already need an IM connection like that for messenger.
Given that information alone, anyone can reconstruct the activity of all their friends into a news feed, and post new content. You'd need a lot of client-side leverage, and the ability to upload to the IPFS network from the client, but all the dynamic information is there.
And there you have a fully distributed facebook clone. More resilient and privacy-sensitive and monopoly-averse than even tools like diaspora and statusnet.
You won't have the full graph that Facebook does, but does that really matter? I suspect the best predictors of who your friends are would be people already in your network.
Not really. There's no reason you couldn't implement Facebook's core functionality with personal servers containing a profile at each node and a pub/sub (or even just pull) system connecting each node with its friends.
https://diasporafoundation.org/
Unfortunately it never really caught on.
All data is encrypted and my server can be hosted anywhere (just as simple as setting up a WordPress instance).
My own "news feed" are a combination of all of the other nodes I'm subscribed to.
My problem with Disapora is that they push the "sign up to someone else's pod" rather than "install your personal pod".
It just feels like a multi-centralised model. Rather than one Facebook with my data, now there are 311 "Facebookesque" Diaspora Pods (as of today).
As much as people hate the idea, a WordPress like (PHP) application as a "personal server" would be successful. There are lots of hosting companies out there where you can get up and running for peanuts.
My aim would be for a simple install:
- Download software and upload to hosting company
- Enter MySQL auth details
- Choose local filesystem or third party (S3, DropBox, Google Drive, MS SkyDrive) with OAuth as the backing store (files stored encrypted).
- Finish
Then I would be able to subscribe to other friends and family personal servers, just like RSS feeds. Friends can access my own personal server through OAuth and leave comments on content. Those comments are echoed back in my private feed which is displayed in their merged news feed on their own app instance.
The technology is already there. It just needs someone who is damn good with PHP!
[0] http://www.mediagoblin.org/
Interesting to think about.
- Is it possible to, say, log into my bank account, see my current balance, and set up bill pay?
- Can I order a pizza over IPNS?
- Can I do realtime chat?
Also, this article says we could use Namecoin and not rely on ICANN for DNS, however, you still need ICANN to distribute ip address to get connected to the internet in the first place. Or am I missing something?
I think that it should be possible to build hybrid centralized/decentralized sites with IPFS, with static public content being fully decentralized and sensitive information going through direct tunnels. Of course, the more decentralized your system the better.
As far as the IP system, yes we are unfortunately still reliant on ICANN being sensible about IP allocations, but the DNS system can be fully deprecated. I do know that there is a project out there to redo the IP system in a decentralized fashion but I can't recall its name right now.
https://github.com/cjdelisle/cjdns
The last one is a no. Well actually yes if you just poll a file, but that's not practical on a large scale. You need an external communication channel to implement that. It's possible to have that be fully distributed as well though, as in tox. You need this external communication channel to implement a lot of things anyway, so it would probably be part of the platform that forms around the idea of a distributed web.
Someone in this thread already asked one of my questions (So is this primarily for static websites?) but my second question is: So is this primarily for personal websites?
I'm having a hard time finding a good way for Facebook, for example, to monetize their website. Targeting ads go out the window with mostly static content. Even more so though, what about Netflix? How is DRM done? How do you make sure only the correct users can access your objects?
edit: Also, doesn't a "permanent web" have an inevitably fatal flaw that you can't free space?
As far as paying a premium over raw bandwidth to access premium content (Netflix), you could pay them to unencrypt it for you to download. DRM though isn't secure in the first place and decentralization won't make it better.
The general idea of decrypting premium content at a cost would work fairly well, however I'm wondering if it's something that can be done easily and dynamically via IPFS. For example, can I, without burdening the network with a billion copies, encrypted with each the client's key, hand that data off to them? If I simply decrypt it and hand it over, it's on a distributed network and anyone can grab it.
The website used to navigate the catalog of premium content could be built as a IPFS page though.
The more people a file is requested by, the more places it is stored. If the file ceases to be requested, and all the participants decide to remove it from their caches, the file is gone.
DHT records are slightly more persistent, but without a sponsor they fade as the network's churn removes the nodes that once hosted them.
It is very hard to delete things, and the most likely things to be deleted are things people are not providing effort to preserve (even one person providing such effort is sufficient to maintain the DHT records.)
In terms of targeting, once you're dealing with "small data" instead of centralised "big data" you need a lot less algorithms to figure out what ads might work. A FB neural net probably does a lot of deep learning to guess that a bunch of mountain-biking friends might be interested in saddles. Decentralised operators could probably figure that out themselves.
While two things may produce the same SHA hash, they may differ with other hashing functions.
What about hashing an object that contains the hashes generated by x hashing algorithms? an extra step of resolution, sure, but some way to easily demarcate that some file experienced a collision with another file, and had to have the number of hashes increased to stave off further collisions?
IE if you have some hashing function with a 1/2^5 chance of a collisions, using that in conjuntion with another hashing function with say 1/2^3 requires that both unlikely probabilities occur, resulting in 1/2^8, where a "stronger" hashing function might only have 1/2^6 (technically stronger than either one of them, but still not as strong as both)
I'd really love some correction on the logic above though, I am by no means well studied in probability or hashing functions, despite some effort, and would really appreciate any corrections.
However, the probability of a hash collision is already incredibly low. As a demonstration of this, here's a list of a tiny subset of bitcoin private keys: http://directory.io/
The human mind just isn't capable of understanding how large of a number the number of sha hashes is.
It's well documented (of course also by the link you posted), the super low likelihood of hash collisions, but I think (hope) the original poster knew that -- was trying to answer assuming that extraordinary case actually happened.
There is no hash function without collisions. The set of inputs is infinite but the set of outputs is finite. (The identity function isn't a hash function--its output isn't a fixed length.)
I suppose this could be made a problem for the user, i.e. if you want to make sure that your page isn't colliding with any other, then simply check first. (Of course there is still a race condition possibility of two colliding addresses being created at around the same time...)
Its generally considered safe to ignore, there are no known sha256 collisions, and most people are fairly confident they wont see one for quite some time.
(We may improve how this upgrade will work down the road by allowing links to link with multiple hashes at the same time, but at this time this has not proved to be necessary, and can be added later)
You can cover earth surface with 8.5×10^36 oxygen atoms. Imagine that every one of them is IPFS file with 256 bit hash. Probability of collision is still less than 0.1%.
If you had the hash of some content, couldn't anyone find all of the ips currently serving it?
EDIT: Found this
https://github.com/ipfs/faq/issues/12
Next, I'd really like someone to implement git backend in IPFS / IPNS. Right now there's https://github.com/whyrusleeping/git-ipfs-rehost but that's just simple hosting.
Nothing is technically stopping me from adding ipns except finding the time to do it.
As ot asked, I am also curious about how this compares to other efforts to create decentralized networks, like Freenet and GNUnet. I definitely plan to pick one in the coming weeks and start using it, hope this catches on.
Still, even after skimming the paper, it sounds like a lot really depends on how well BitSwap is implemented. At one point, the paper says this:
> While the notion of a barter system implies a virtual currency could be created, this would require a global ledger to track ownership and transfer of the currency. This can be implemented as a BitSwap Strategy, and will be explored in a future paper.
So it kind of sounds like they need something like bitcoin, but don't want it to be tightly coupled to bitcoin - which is probably smart. But without a single clear solution for a reliable BitSwap strategy (and I'm not sure how it could be reliable without a distributed global ledger of some kind), it's hard to see how/why the necessary resources will be contributed to the IPFS network.
Really cool stuff by the way - I'm very excited to see people working on this, and I hope IPFS (or something like it) is successful.
One of the first questions that came to me: Is there any way to add a forward error correcting code, like an erasure code to IPFS? I didn't find any discussion of this in the IPFS paper.
This seems somewhat critical to be able to compete with modern centralized storage systems which are likely to use FEC extensively to provide the level of redundancy that customers expect. Modern FEC codes can provide phenomenal levels of redundancy with less than 50% overhead. IPFS seems to rely on many fully-redundant copies?
From a practical standpoint, I'd like my local node to have a full copy so that my friends coming over don't need to reach out to the (maybe offline) internet but I don't see why we shouldn't support encoding the chunked blocks in such a way?
Though perhaps I'm misunderstanding your question - if you want multiple IPFS nodes to coordinated on storing parts of an erasure encoded file? that would be useful too. Again I think it's possible - you could build a small distributed system to host files this way. The most interesting variant though is if IPFS could have support for something like this natively - so nodes run by different users can each pitch in a little. Conceptually it's possible, but probably the protocol doesn't support it.
Reed-Solomon ECC are just amazing!
I imagine getting the static html/js/css for the client-side app onto IPFS should be simple enough, but how would I handle bootstrapping?
Even in a decentralized service, clients need to connect to a seed server somewhere to start querying for other peers on the DHT. Would it be possible to host such a seed server on IPFS? Or would the seed server still need to live on a traditional server somewhere on the "old web"?
Since IPFS runs on its own DHT, are there any plans for a client-side IPFS DHT library for apps to use to connect to peers on the IPFS network? Similar to https://github.com/feross/bittorrent-dht, but with native support for IPFS peers?
Something like this would make building a client-side real-time app based on IPFS a much more streamlined experience since developers won't have to depend on an external DHT.
Many users want to do things with other users
That means there has to be a way to persist state and history (of a group activity, an evolving resource, etc.) across machines. Mental poker is still very hard to do for groups. Since most clients will not have amazing uptime, that means you still need a network of servers. Servers that manage security behind a firewall.
So, the conclusion is -- for group writable data, the best you can do is have servers be completely interchangeable, but you will still need servers and networks of servers.
The good news is, many of the challenges are orthogonal to each other. Someone's building a Mesh network, for instance. Someone else is building the protocol layer with IPFS. We're building the social layer for this coming web.
Take other patterns from clojure immutable data structures for various other kinds of group edited time travelling sites
as for conflict resolution, merging, etc, it kind of looks like a (mostly) solved problem. I don't see how conflicts could occur in a comments stream. It's append-only.
So yes, for append-only tree structures, this may work. In terms, of say, a Google Doc that means you can never delete or insert any text, but only append text to the end of paragraphs etc. Certainly very limiting.
Unless there's something you haven't described, in which case I encourage you to provide a more in-depth comment describing a solution for actually inserting or modifying the tree.
http://project-maelstrom.bittorrent.com
I've worked with IPFS, and it's really amazing. The development on it is very active, and the main developers are easily accessible for questions on Freenode#ipfs. I've been trying to build a tiny consensus protocol on top of it, but its native support for signing is still a little rough around the edges (and of course I've been distracted by a thousand other side-projects).
- Now I want to view a whole web page. Is that possible?
- Let's say I have a site that I want to publish through this, along with a bunch of files (images, downloads). I'm serving that site now through Apache. Do I need to 're-publish' each file (using the command in the article) every time a file on there changes? Or is there some automated way to have a distributed version of the latest version of my site? I mean, it's fine and dandy that people can store cached copies, but what I want is that if there's an update to my software, that people actually get the latest version when they click 'download', not that they get an old version because some dude somewhere made a mistake at one point 'hosting' a static version of my site and not updating, and visitors just happening to use that version because that dude is closer to them (in network topology terms) than I am.
You can use it like a unix command line tool to add and get files but you can also run it in daemon mode. It then also opens a HTTP server locally which allows you to use the same URLs as above just with http://localhost:8080 as the schema and host.
2) You can use the ipfs cli tool to add a directory with your sites html and other static content. The client will then give the the root content hash. Then you open /ipfs/$rootHash/index.html or /site2.html and voila! :)
Btw, this article is serving from ipfs. Try 'ipfs get QmNhFJjGcMPqpuYfxL62VVB9528NXqDNMFXiqN5bgFYiZ1' and take a look in that directory on your harddrive. Also: when you now run 'ipfs daemon' your node will help with requests for this content.
3) To update content with a static name, you can use IPNS, it's also mentioned in the article. IIRC: To resolve those entries (and get the latest ipfs content hash) the node with the corresponding key for the ipns entry needs to be online. If it isn't, you get an error like 'Path Resolve error: could not resolve name.'. Maybe come to the #ipfs irc channel on freenode for more specific questions.
Distributed content in IPFS is eventually purged from nodes unless one of the hosting nodes has pinned it to be retained. Therefore, if no-one at Time x views certain content and pins it, then unfortunately at Time x + n that content might disappear just as thoroughly as under HTTP.
Unfortunately I fear that means that 'popular' content persists whilst niche and unique data might still fade away.
... and it also sounds like usenet, but here everybody is a news server and they selectively host articles.
I wonder if this would also be possible in this case?
For example, my small server might serve 200 sites but I'll backup around 500 or so older ones somewhere else.
Perhaps it is also possible to tweak the caching algorithm to keep track of files that are rare on the network.