Ask HN: Can we create a new internet where search engines are irrelevant?
If we were to design a brand new internet for today's world, can we develop it such a way that:
1- Finding information is trivial
2- You don't need services indexing billions of pages to find any relevant document
In our current internet, we need a big brother like Google or Bing to effectively find any relevant information in exchange for sharing with them our search history, browsing habits etc. Can we design a hypothetical alternate internet where search engines are not required?
397 comments
[ 2.9 ms ] story [ 275 ms ] threadFor routine stuff i tend to have established resource starting points, like documentation, official/community sites, blogs/news feeds, and yes: link directories (like awesome lists).
Most web sites then also had a healthy, sometimes surprising link section, that has all but disappeared these days.
This is what I did back in 2015 as a project to increase my SEO rank to my business. Basically spam directory (and create my own) just to increase my pagerank.
it sounds like what you really want is a decentralized search engine and anonymous by default as apposed to no search engine.
[1][https://www.trbimg.com/img-5320a78f/turbine/orl-0312aol-1996...]
I think this gets boring quick...
I guess that would be the age of smaller communities centerd around a few websites only? Maybe, I don't know if we can consider google as enabling a real global community as of today. I pretty much browse around the same websites. Anything I want to find without a precise source of information in mind, I use google and stumble upon ads and ads and sometimes ads, but rarely an answer.
I sometimes still search stuff manually browsing through websites indexes. Some things are difficult to find with keywords. Equations of which the name you forgot. Movies with a plot so generic billions of result would be associated with it on a search engine. That piece of music of which you could write the notes on a sheet but don't remember the title.
The question is whether this would work in an adversarial setting where every party tries to inflate their page rankings by any trick they can find.
Uh...what? How do you define this?
https://libraryofbabel.info/search.cgi
This is not simple, and your Ask HN reeks of ideology and contempt without so much as an inkling of the technical realities that would have to be overcome for such a thing to happen. That goes for both old and new internet.
/rant
I don't think this question belittles Google's work.
I feel saying that would be like saying that animals that chose to live on the land were belittling millions of years of evolution in the water.
People working at Google chose to spends their time building a search engine for the world wide web, fine. That does not mean that sharing information accross a network has to be done via { world wide web, google }.
All of this is purely theroical of course, but I'm sure someone more creative than me would find another solution. Maybe not a solution that would exactly fit OP's description, maybe not a solution that would be practical with the current infrastructure.
But a solution that would render Google as-is obsolete ? Yes, I think that would be possible.
If we could kill advertisement permanently, we can have an internet as described in the question. This will almost be like an emergent feature of the internet.
https://hackernoon.com/wealth-a-new-era-of-economics-ce8acd7...
- ranking content that users you have upvoted higher
- ranking content that users with similar upvote behaviour higher
While there is a risk of upvote bubbles, it should potentially make it easier for niche content to spread to interested people and make it possible for products and services to spread using peer trust rather than cold shouting.
That's how you make echochambers
I was also wondering what would be good options to store votes/upvotes in a decentralized way.
Yeah, I wonder if there is a cheap way to test this. Actually! There could be! Like using favorite's here on hacker news. That could be mined and visualized in various ways. (Although a quick sample shows me that it's a rarely used feature)
> I was also wondering what would be good options to store votes/upvotes in a decentralized way.
Yeah there are a lot of interesting optimization challenges if you really want to utilize upvote graphs for ranking.
This is what Reddit originally tried to do before they pivoted.
https://www.reddit.com/r/self/comments/11fiab/are_memes_maki...
Makes me think that their original plan could still work if they just put a bit more effort into crafting that algorithm.
For example, the main criticism brought up is that things that you dislike that your peers like keep getting recommended. Why not add a de-ranking aspect into it and try adding downvote-peers in addition to upvote peers.
I imagine you could create this interesting query language that could answer questions like: what things do you like if you like X and Y but not Z? (I kind of remember that something akin to this have been hacked together using subreddit overlap.)
I sure hope my content of preference beats out yours for not getting killed.
Can you propose any viable alternative?
https://contributor.google.com/v/beta
So from a user's perspective it didn't fully work. Also the ad space wasn't fully removed (perhaps due to technical reasons) but was replaced with a blank image. It also didn't catch on much.
So they tried to pivot and now the program works with certain cooperating websites to fully get rid of all ads but I'm sure bigger websites would rather be in total control of monetizing themselves and can spend on the necessary IT infra. similar to most online newspapers these days.
I think an advertiser (eg. a legal firm) might be willing to pay eg. $10 per ad impression but no user is willing to outbid it so I think the first model (outbid in the auction) is more sustainable and profitable for both parties but needs to have all ad exchanges on board.
So in short, it's been tried but wasn't an instant (or even a slow) success and idk whether Google will continue investing in it or not.
But I think the combination of advertising+search engines is particularly bad, so paying for search would be a great first step.
By and large, people don't seem to be willing to pay for content on the web. Hence, advertising became the dominant business model for content on the web.
Find another way for someone to pay for relevant content and you can do away with advertising. It's as simple as that.
Not so simple. What is relevant for me may be irrelevant for you.
I don't think the causality is right here. People might not be willing to pay for content on the web because advertising enables competitors to offer content for free. If you removed that option, if people had no choice but to pay, it might just turn out that people would pay.
Isn't this what different newspapers like NYT and WSJ are moving towards? Why can't both models coexist?
Slave labour, selling poison or dumping waste into rivers are all superior business models too, but that doesn't mean they should exist in a civilized society.
Just because it totally destroys another business model doesn't mean it is wrong. Felony interference with a business model protectionism isn't good for societies. Historically this stagnant "stability" gets them lapped and forced into the modern world if lucky or conquered if not no matter how vigorously they insist that it is the only and right way.
--
[0] - as seen today; not the imaginary "informing customers about what's on the market" form, but the real "everyone stuck in a shouting contest of trying to better manipulate customers" form.
There absolutely are paid options on the web. It's just that they don't seem to appeal to a sufficient number of buyers so advertising could become irrelevant.
Yes.
> There absolutely are paid options on the web. It's just that they don't seem to appeal to a sufficient number of buyers so advertising could become irrelevant.
They aren't appealing in the presence of ad-subsidized free alternatives. Remove the latter, and they just might become appealing again.
For example, using browsers that impose a Content Security Policy that prevents anything from being loaded from domains other than the origin.
You can block third party advertising structurally using uBlock without ruining the internet for everyone else.
I think a combination of consumer protection laws, truth in advertising laws and data protection laws, all turned up to 11 (even GDPR), could achieve most of the desired outcome on the Internet without much problematic "content-policing". But I'm not sure. You won't eliminate advertising from the Internet entirely, but making it illegal would make undesirable advertising more expensive, by creating vast amount of risk for advertisers and simultaneously destroying the adtech industry, thus rendering most of the abusive practices that much less efficient.
(Also, to be clear, I want all advertising gone. Not just on-line, the meatspace one too.)
For the remaining free sites you will see advertising in different forms (self promotion blog, the upsell, t-shirt stores on everysite, spam-bait).
Advertising saved the internet.
Now tracking.. for advertising or other purposes is the real problem.
Until then, you're going to have demand for ferrying information between sellers and buyers, and vice versa, because of information asymmetry. You may disagree with some of the mediums currently used, finding them annoying, but advertising is always evolving to solve this problem, as is evident in the last three decades.
There's a saying in sales: "people hate to be sold, but they love to buy"... which is akin to what you are saying here. Advertising isn't the problem... the problem is that the reasons why people are promoting aren't novel enough... (rent seeking... which creates noise)
https://en.wikipedia.org/wiki/Project_Xanadu
It's a vision for an academic, small scale, network, not for a viable global web.
With a distributed open search alternative the algorithm is more susceptible to exploits by malicious actors.
Having it manually curated is too much of a task for any organization. If you let user vote on the results... well, that can be exploited as well.
The information available on the internet is to big to make directories effective (like it was 20 years ago).
I still have hope this will get solved one day, but directories and open source distributed search engines are not the solution in my opinion unless there is a way to make them resistant to exploitation.
ODP/DMOZ worked quite well while it was around. I don't think it would work equally well nowadays as a centralized project, because bad actors are so much more common today than they were in the 1990s and early 2000s; and because the Internet is so astoundingly politicized these days that people will invariably try to shame you and "call you out" for even linking to stuff that they disagree with or object to in a political sense (and there was a lot of that stuff on ODP, obviously!). But federation could be used to get around both issues.
I feel like manually curated information is the way to go, you just have to find some way to filter out all the useless info and marketing/propaganda. You can't crowd source it because it opens up avenues for gaming the system.
The only solution I can think of is some sort of transitive trust metric that's used to filter what's presented to you. If something gets by that shouldn't have (bad info/poor quality), you update the weights in the trust network that led to that action so they are less likely to give you that in the future. I never got around to working through the math on this, however.
This trust (weighting) should be able to propagate as a (semi-)transitive property throughout the network to take advantage of your trusted peers' trusted peers. This trust weight propagation would need to converge, and when you are served content that has been labeled incorrectly ("high-value" or "trustworthy" or whatever metric, when you don't see it that way), then your trust weights (and perhaps your peers') would need to re-update in some sort of backpropagation.
The hard part is keeping track of the trust-network in a way that is O(n^c) and having the transitive calculations also be O(n^c) at most. I'm quite sure there are ways of doing this (at least with reasonably good results) but I haven't been able to think through them.
You're just shifting around your trust problem. You need to handle 4chan level manipulation (million of users coordinating to manipulate polls), or Scientology depth (getting thousands of people in to USA government jobs in order to get recognised as a religion). If it's "we'll catch it in moderation" then whoever wants to manipulate it just gets a moderator ...
"Super-moderation": will a dictatorship work here? I don't see how.
"Meta-moderation": you're back to bad actors manipulating things with pure numbers.
But think of how we solve this problem in our personal interactions with other people, and this should be a clue for how to solve it with computational help. We have a pretty good idea of which people are trustworthy (or capable, or dependable, or any other characteristic) in our daily lives, and based on our interactions with them we update these internal measures of trustworthiness. If we need to get information from someone we don't know, we form a judgement of their trustworthiness based off of input from people we trust--e.g. giving a reference. This is really just Bayesian inference at its core.
We should be able to come up with a computational model for how this personal measure of trustworthiness works. It would act as a filter over content that we obtain. Throw a search engine on top of this, sure, but in the end you'd still need to get trustworthiness weights onto information if you want it to be manipulation-resistant. This labeling is what I mean by manual curation. You can't leave that up to the search engine or the aggregator because those can be gamed, like the examples you gave for aggregators and SEO for search engines have shown.
We really don't. People get surprised all the time that someone had an affair, or cheated, or ripped someone off, or whatever. "But I trusted you" ...
It's actually relatively easy to fool people in to trusting you, as many red team members will probably confirm.
Look at someone like Boris Johnson, people are trusting him to lead the country knowing that he's well known to betray people's trust and that he even had a court case lodged against him based on his very blatant lying to the entire country. You can even watch the video of him being interviewed where the interviewers says (paraphrasing) "but we all know that's a half truth" and BoJo just pushes it and pushes it and refuses to accept that it's anything other than absolute truth.
>If we need to get information from someone we don't know, we form a judgement of their trustworthiness based off of input from people we trust--e.g. giving a reference. //
This is domain authority again - trust some domains manually, let it flow from there. If that domain trusts another domain then they link to it, trust flows to the other domain, and so on. Maintaining such trust for a long time adds to a particular domains trust factor, linking to domains not trusted by others detracts from it.
>This is domain authority again - trust some domains manually, let it flow from there. If that domain trusts another domain then they link to it, trust flows to the other domain, and so on. Maintaining such trust for a long time adds to a particular domains trust factor, linking to domains not trusted by others detracts from it.
This can be gamed if you're able to update the trustworthiness of a domain for other people, and that's why a trust metric needs to be mostly personal, and should update dynamically based on your changing trust valuations.
Seriously, I'm not so sure -- I try to trust first and then update that status as more information becomes available; but that's more of a religious position.
I don't think it's necessarily instructive to look at my personal modes here. I guess my main point is that if you're going to say "well humans have cracked trust, we'll just model it on that" then I think you're shooting wide of the mark.
But you want 'manually curated' but not 'crowd sourced', which suggests you want an individual to or small group to find, record, and curate all pages (? or domains, or <articles>, or ...) across more than 60 Billion pages of content??
There's something like 1000 FOSS CMSs - I would be surprised if there's a million domains with relevant info to sift through just for that small field.
There's no way you're curating _all_ that without crowd sourcing.
Of course you don't have to look at everything to curate, but how are you going to filter things ... use a search engine?
I think it's not possible as domains don't have to be registered necessarily - a server can serve a domain at a particular IP so long as the requesting client uses a domain that the server responds to.
Obviously if it's registered domains then you can in theory just get the list from the registrar. They probably sell the full list for a price.
I imagine you can harvest a list with sufficient resources.
The evil big brothers may not be necessary. We just need to expand alternative search engines like YaCy.
1. Finding information is trivial
2. You don't need services indexing billions of rows to find any relevant document
Maybe I misunderstand your proposal but to me, this is not technically possible. We can think of a modern search engine as a process that reduces a raw dataset of exabytes[0] into a comprehensible result of ~5000 bytes (i.e. ~5k being the 1st page of search result rendered as HTML.)
Yes, one can take a version of the movies & tv data on IMDB.com and put it on the phone (e.g. like copying the old Microsoft Cinemania CDs to the smartphone storage and having a locally installed app search it) but that's not possible for a generalized dataset representing the gigantic internet.
If you don't intend for the exabytes of the search index to be stored on your smartphone, what exactly is the "on-device search agent" doing? How is it iterating through the vast dataset over a slow cellular connection?
[0] https://www.google.com/search?q="trillion"+web+pages+exabyte...
We already have the means to execute arbitrary code (JS) or specific database queries (SQL) on remote hosts. It's not inconceivable, to me, that my device "knowing me" could consist of building up a local database of the types of things that I want to see, and when I ask it to do a new search, it can assemble a small program which it sends to a distributed system (which hosts the actual index), runs a sophisticated and customized query program there, securely and anonymously (I hope), and then sends back the results.
Google's index isn't architected to be used that way, but I would love it if someone did build such a system.
Though to your point, google probably ends up storing this information in the cloud
In the simplest case, you could make a search engine in the form of a big, public, regularly-updated database, and let users send in arbitrary queries (run in a sandbox/quota environment).
That's essentially what we've got now, except the query parser is a proprietary black box that changes all the time. I don't see any inherent reason they couldn't expose a lower-level interface, and let browsers build queries. Why can't web browsers be responsible for converting a user's text (or voice) into a search engine query structure?
Apple uses local differential privacy to help protect the privacy of user activity in a given time period, while still gaining insight that improves the intelligence and usability of such features as: • QuickType suggestions • Emoji suggestions • Lookup Hints • Safari Energy Draining Domains • Safari Autoplay Intent Detection (macOS High Sierra) • Safari Crashing Domains (iOS 11) • Health Type Usage (iOS 10.2)
Found via Google...
I'd love to be able to configure rules like:
+2 weight for clean HTML sites with minimal Javascript
+5 weight for .edu sites
-10 weight for documents longer than 2 pages
-5 weight for wordy documents
I'd also like to increase the weight for hits on a list of known high quality sites. Either a list I maintain myself, or one from an independent 3rd party.
Once upon a time I tried to use Google's custom search engine builder with only hand curated high quality sites as my main search engine. It was to much trouble to be practical, but I think that could change with an actual tool.