HN Explorer

Ask HN: Is there an opt-in search engine for the World Wide Web?

1 points by CaptainFever ↗ HN
Related to previous articles about Fediverse/Mastodon search[1][2]. Assuming that one comes to the conclusion that ethical search engines should be opt-in, and that there is no ethical difference between Fediverse and web search engines, I'm curious if there are any efforts out there to create an opt-in search engine for the web. Such a search engine would not index any pages, unless they are submitted directly, or have a machine-readable attribute (e.g. robots.txt, "yesindex") that explicitly allows indexing.

I'm also open to objections to either the former or latter assumptions, though that is slightly off-topic.

[1] https://news.ycombinator.com/item?id=34221851

[2] https://news.ycombinator.com/item?id=34400143

2 comments

[ 5.5 ms ] story [ 12.7 ms ] thread
verdverm
Most people want their content discoverable so defaulting to indexing would seem preferable as well. How would a search engine get people to opt in? Why would web admins take the time to add a fringe search engine crawl requirements to their site? Basically the search engine you describe would have a really hard time getting started
CaptainFever
That's very true. Such a search engine would require that it is not fringe, but this is a catch 22 situation.