Ask HN: Whatever happened to exploring the internet?

320 points by mickjagger ↗ HN
I seem to have collapsed down to checking 5 to 6 sites.

Where would I even go to find other sites that might be of interest?

I can’t even really think of any other sites to visit.

Whatever happened to the idea of “exploring the Internet”?

284 comments

[ 4.3 ms ] story [ 219 ms ] thread
When I am in that type of mood I just head to my huge uni library and randomly pick a book. No unnecessary jabbering from the chimp troupe to deal with, and no wading thro all sorts of scams, advertising, self promotion etc. Todays internet just doesnt match a nice library.
If I could give you a thousand upvotes, I would. I have spent countless hours in libraries exploring the stacks, absolutely everything was at my fingertips. Nothing can compare.
This is why I was so excited in the early days of Google Books. The digitalization of most written human knowledge would have created a modern Library of Alexandria, a sprawling, perfectly catalogued archive. An open API would make it so you could walk through in a metaverse platform, or scroll through by section, shelf, and spine. A bot librarian could guide blind users through and read books for them, either via BrailleNote or voice assistant.

Instead, copyright turned Google Books into a worse version of Amazon's "look inside" feature, depriving the internet of immeasurable substance.

Planets and webrings are still around, and are a valuable source. Other forums and message boards like HN, lobste.rs, tilde.news, lemmy.ml and reddit.com are also still doing well.

EDIT: IRC/XMPP/Matrix chatrooms are still doing well, too!

> I seem to have collapsed down to checking 5 to 6 sites.

I check about 20 sites regularly even when I have other interesting things to do.

> Where would I even go to find other sites that might be of interest?

Search engines?

> Whatever happened to the idea of “exploring the Internet”?

It sounds like you would like to find lists of recommended sites that people used to have on their homepages. Most people don't maintain such lists anymore, because 1) nobody reads them 2) they are busy pushing updates and consuming those of other people. Most people find more interesting stuff than they have time to consume already. (Who even needs entertainment industry anymore?)

Also I don't frequent social media, unless HN counts.

Edit: Sometimes when I want anything to read, I search "is java dying" or "why c++". There has always been something new, albeit I do it only two or three times a year.

> Search engines

Which search engines in 2021 give you real, interesting websites and not SEO gamed content pages that exist only to serve ads?

Bubbling can be avoided with something like DuckDuckGo or Startpage, albeit SEO gamed content is still an issue.

Marginal search engines may be better at avoiding it, as optimizing for them wouldn't be that profitable, but I haven't tried any.

You could try checking out lower results, or search the SEO gamed content for hints that help you dig deeper.

> Which search engines in 2021 give you real, interesting websites and not SEO gamed content pages that exist only to serve ads?

You could try https://searchmysite.net/ which only includes user-submitted and moderated websites, and tries to keep out spam by detecting pages with adverts and heavily downranking them in the results.

I honestly do think link pages should make a comeback. Websites are really bad at linking to each other. It's like they are trying to trap their visitors or something and prevent them from leaving by having no exits, but no visitor likes to feel trapped like that.

It's fun to assemble link lists, it has a sort of scrapbooking quality to it, and as long as the links are of high quality, they can be very enjoyable to browse through as well. That's the benefits beyond the service they provide for the personal website ecosystem which is really struggling with serious discoverability issues.

I agree in a way, but for different reasons.

The Internet can host public debate where everyone can participate equally well in a way that was never before possible, whereas mainstream media is mostly a relic from a time when writing was ink on paper, and publishing was therefore expensive.

People could commend blogs and articles they think you should read (and publish their own too!). They should be able to do so without fear of losing friends or being kicked from the platform when they find something objectionable in them.

Blessed is he, whosoever shall not be offended in me. -- Jesus the Christ

But the tongue can no man tame; it is an unruly evil, full of deadly poison. -- James 3:8

I was going to say that the death of StumbleUpon really closed off whole portions of the web for me. Their algorithms were really top-notch and the replacement service isn't worth checking out.

Does anyone know why they killed off StumbleUpon? It was probably the best part of 2005-2015 for me.

Edit: I like your link but it seems to lack the interest-targeting algorithms that SU used to have.

I cite my comment from Reddit I posted some time ago explaining what happened to SU:

> Stumbleupon had a declining user base because more and more spam flooded the site. Moderation couldn’t just keep up to it. As a result, advertisements were increased to keep up the profits from the remaining users. As you might imagine, this pushed those remaining users off the site even more. At some point, the management decided to put effort into a new platform: Mix.com. The CEO also spent a lot of his time building Uber (yes, the taxi company).

You might get a kick out of this website: https://usefulinterweb.com/ It curates a list of 1000s of interesting websites, and adds about 3-4 new links daily
I found the idea of this promising, but the first link I followed took me to an Amazon page to buy a book. I feel like that's directly taking me out of the useful internet experience, as I was expecting content on the topic.
"Web 2.0" was about monetizing everything you could visit that might be remotely interesting, and separating you from the content with a paywall. We're now looking into "Web 3.0" which will bring even more concepts of monetization, ownership and client-side computation. It's a mixed bag going forwards, but 'exploring the web' died with Web 1.0 as far as I'm concerned.
I've put some thought on this topic, and I think the driving force between the Internet seeming so small nowadays is a combination of changes to how search engines work, as well a move from forums to big social media which has meant a shift from organic community discovery to being drip-fed content from based on what an algorithm thinks will be engaging.

The Internet, as you remember it, still very much exists. Some forums have shut down, but there are still small personal websites, blogs, all that stuff. They're just really hard to find with Google and facebook/reddit/twitter.

Here are some cool and creative things I've discovered recently. I have no affiliation with these projects, I just thought they were cool:

http://www.lileks.com/

http://dreamcult.xyz/

http://sod.jodi.org/index.html

http://godxiliary.com/

https://www.floppyswop.co.uk/

https://www.dedware.com/

Lileks... I've been visiting his site off and on since the late 90's. Always something interesting there.
people always go towards laziness and convenience, that seems to be a axiom of life.
I'm not sure I'm convinced about that. The desire for convenience may be a widespread cultural message, but I wonder if it comes from people and not marketers. A lot of people seem to suffer in perpetual convenience, they may not be able to articulate it, but it makes them uncomfortable. After a while they feel restless and crave some sort of meaning in life, something to engage with, an interesting idea, a hobby, a project. Some form of work, not for salary but to create something or learn something or do something. Something real and authentic.

This thread is actually a good example of that, and OP isn't the only one who has noted that the Internet seems to have turned into a soulless mall where nothing has any effort or thought put into it. Well, it hasn't, but when you engage with the Internet through certain tools it sure looks that way.

i agree with you wholeheartedly, nice point. I guess you can liken it to junk food, most people would be happier and healthier without it but marketing and people's propensity for an easy way out if it exists has lead to people being fat, unhappy and addicted to junk food. I wonder who is to blame? People for the choices they make, or businesses who know people's propensity to do stuff even though it goes against an individuals own best interests.

b/w i found your search engine and blog, you are exactly the type of people we need more of, thanks for everything!

> I think the driving force between the Internet seeming so small nowadays is a combination of changes to how search engines work, as well a move from forums to big social media which has meant a shift from organic community discovery to being drip-fed content from based on what an algorithm thinks will be engaging.

I think you hit the nail on the head. Google used to prioritize forum posts in their search results. They even had a feature to limit search results to only forum posts via a "Discussions" option on the results page.

These days, Google prioritizes social media content, click bait and blogspam that has a lot of ads. The "Discussions" option is gone, and even recent forum posts are nowhere to be found on the first few pages of search results.

A similar thing has happened with Google News. It used to be that News was a search engine that basically told you all the latest blog posts and discussions on a topic, rather than trying to provide capital-N News. I was recently looking for that functionality to find discussions on an obscure topic and it's completely gone.
The recommender algorithms cater to the dumbest common denominator, i.e. content with the broadest appeal (no niche stuff) to the most active internet users (people with way too much spare time).
Recently I began "exploring the internet" again with the help of a niche search engine that was posted here on HN not that long ago [1]. Seems like it returns lesser known, interesting, contrarian websites where the authors speak their mind. For example I found websites that: speculate that milk and lactose might be a hidden cause for several chronic conditions [2]; try to re-explain evolution by re-positioning the relationship between life, organism, and DNA [3]; rich homepages like this one [4];

[1]: https://search.marginalia.nu/ [2]: http://www.nomilk.com/ [3]: https://bwo.life/org/index.htm [4]: http://www.valdostamuseum.com/hamsmith/TShome.html

A lot of this was painful. I'll comment on bwo.life, since it is somewhat close to some of the things I know a little bit more about.

> Despite all this life and death, I doubt whether anyone would be tempted to describe the embryo’s cells as “red in tooth and claw”. Nor do I think anyone would appeal to “survival of the fittest” or natural selection as the fundamental principle governing what goes on during normal development. The life and death of cells appears to be governed, rather, by the developing form of the whole in which they participate.

This is precisely the sort of BS you get when you do not participate actively in a field, but instead go off on the side to live in your own world. During development, suvival of the fittest is very much at play; we have evidence for this, for example in the developmental trajectories of stem cell niches. Cells outcompete each other, and make it difficult for other varieties of cells to "live" along side them (this does not necessarily result in apoptosis, it can also cause re-differentiation in the less "successful" cells). A crucial reason for this is error correction: sometimes, new cells are defective in various dangerous ways (e.g. cancerous, or have problematic genetic information), and "survival of the fittest" helps to error-correct for this, as usually such defective cells are "less fit".

What's surprising here is that cells can change their fitness, "on purpose", in ways that do not make evolutionary sense. They might be programmed to reduce their fitness, or entirely explode, if they receive certain inputs. So, clearly this is an emergent process where we must take into account cellular programming along side "naive" natural selection.

"Old hat" to biologists, really, in that they have been studying this. They're studying it until today. New things are discovered all the time. There is no sense of certainty, only that there are some general outlines emerging.

Coming back then to this article: saddening, if not disgusting in its pretentiousness. Link [4]is even more depressing. I don't know if people understand how dangerous quackery is: it destroys lives. Most importantly, it destroys the lives of children whose parents fall into this.

The broader internet (being difficult to explore given how modern internet ecologies box us in) is thankfully not made up primarily of this kind of..."garbage", but it is often painted as being that way.

So, for anyone who came across this: please don't think the un-popular internet is a cesspool of "people speaking their minds, in a convenient vacuum". Rather, there are websites where people who are humble about their understanding, even if it is substantial compared to the average person's. For example: John Baez's homepage (https://math.ucr.edu/home/baez/) is far richer than link [4], and actually gives you useful knowledge.

+Fravia's lore on internet searching (and reverse engineering in general) are masterpieces; showcases of what the internet can produce, while also working as effective vaccination against disinformation: http://biostatisticien.eu/www.searchlores.org/indexo.htm

His strategies for how to search for useful information on the internet remain relevant today.

Well what do you propose, should search engines be arbiters of truth? That would give them a terrifying power over society at large.
Regarding website [4] - I agree with you, I just showed it as an interesting example of what might be found.

But regarding bwo.life, I think you are missing the forest for the trees. I read quite a lot of articles on that site and your comment in no way disagrees with anything written there. Honestly speaking - I got the impression you just wanted to highlight some details you happen to know about development due to your field of work. And you disagree in the first paragraph, saying that there is some natural selection between cells. But then, it seems obvious that survival of the fittest cells cannot explain development alone, so you backtrack and say that this is not the full story while somehow remaining in disagreement with the author. I don't get it.

Scientists have never claimed that survival of the fittest is sufficient for development. bwo.life sets up a strawman to tilt against. A waste of time.
I'm using rss/news reader Feedly.com to follow > 100 sites daily, most entrys are "marked read" others saved for more close reading. The list of sites expand weekly by interesting links from sites like hacker news &c.
I think it's important to ignore the commercial part of the Internet when discussing this, sure, back in olden days, commercial sites interlinked a lot more than they do now, but they had to, now they don't.

There's something satisfying in browsing some site, and following a link to the next.. That's really what I think of as "exploring". Exploring only works when you link to your friends sites (and when your friends has sites you can link to) and they link back.. The interlinking has probably weakened somewhat, I don't know why, but I suspect it to be partly because we've gotten so used to link rot, and nobody wants to have a site full of dead links..

That said, I still link to others. And I must shamlessly plug https://geekring.net/ as a tool for exploring (though, it's only about a hundred pages), you could add yours! :)

When I was first using the web, it was largely via Yahoo! directory [1] results. Most of the interesting sites were people's personal sites (with lots of URLs like something.edu/~user), and most people maintained a list of links to other sites they found interesting -- chances are if you liked their site, you'd like most of the ones they linked to as well.

Pretty much all that stuff is just gone now.

The closest thing I do like this today is probably Twitter: follow someone interesting, and you'll start discovering people/projects/sites/etc they find interesting. Similarly, HN/Reddit are maybe the closest thing to randomly browsing the "directory", but otherwise everything is via organic search results for a specific topic.

[1] https://en.wikipedia.org/wiki/Yahoo!_Directory

As the Internet grew, businesses built themselves around it. Metrics optimized websites for behaviors that aren't organic. You only check 5 to 6 sites because, either unintentionally or intentionally, for better or worse, they indulge you in ways that the old Internet did not.

Look for communities that are built around the humans in them, vs a single company. Web rings, Gemini, IRC are all great places to start.

>Where would I even go to find other sites that might be of interest?

Hacker News?

Or Twitter, at least half of my feed consists of people who frequently share links to a variety of tech-related sites. Also many people have an interesting site linked in their bio. (If you prefer just your feed without Twitter's ads and suggestions try https://tweetdeck.twitter.com).

Odd thing to post on Hacker News -- a website dedicated to links to all different kinds of websites.
Well for starters, we used to call it "Surfing the Internet" not "exploring the internet".

The modern web was designed to rope you in and keep you there. Look up random stuff that interests you, not using a big name search engine. Then if you find a site on a webring, and look at connecting sites.

A lot of the original metaphors for "using" the internet carry connotations of exploration or perusing, but they've been in use for so long the meaning has sort of gotten worn off.

Listen to these words and think about what they actually mean: Internet Explorer, or Web Browser, even Netscape Navigator with its nautical theme has such connotations.

You can add konqueror, and safari to this list
You mean Internet Explorer isn't for exploring the internet!
Haha I'm sure it was "surfing the web" ... I never heard anyone say "surfing the internet"
You're right, it was never internet. For me it was net or web, but it was always surfing.

A little further back it advertising called it the "Information Superhighway"

You might get a kick out of watching old NetCafe or Computer Chronicles episodes on archive.org, fellow netizen. Did anyone ever really use that term? I like to playfully use it after watching NetCafe episodes a few years ago.
Here's a tortious causal chain that I think explains everything.

Computing evolved up the point of Multics. The military has always been a driver of computing research to some extent. The deployment of computing resources to help plan airstrike missions showed a critical need for developing a system in which a single computer could handle multiple levels of secure data. The research resulted in capability based security, which was in the process of being folded into Multics.

The folks at Bell labs happened to have a spare DEC machine, and having seen the complexity of Multics, decided to eschew capabilities, and instead relied on a much simpler, and quicker to implement system based on group and user IDs into Unix. This quickly spread to be the defacto multi-user model of security across the academic world.

Over time, PCs came to dominate the low end of computing. When it came time to implement multi-user and network systems, the Unix model, or a slightly upgraded model, based on access control lists (as in Windows) effectively ate the world.

Eternal September happened, and the internet went commercial. With this, we now have persistent internet, and are stuck with the oversimplified security model from Linux and Windows dominating everything. As such, no computer is actually secure.

Because computers aren't secure, you can't trust programs that run on them to be secure. Because of this, you can't trust the web browser on your computer to not get you into trouble if you click on the wrong link. This results in a very strong tendency to avoid clicking on links from unknown domains and sites among the general public.

Because the audience has settled into a few walled gardens, most of the authors of content have had no choice but to move to do the same.

And here we are, because capability based security is seen as too complicated (it doesn't have to be, in fact it can be simple to use), we're all stuck with facebook, twitter, etc.

I generally don't like to bring any attention to language, but I do wonder if you mean tortious.
Yeah, I really don't think you should (and neither should I), since we're never as smart as we think we are.

The author did not mean tortious, but instead tortuous. But one immediately gathers this from the context anyway. Who cares what the spelling is?

"Tortious" is just as self-deprecating in that sentence. Is it an unfair slight to identity-based security systems, or a overlong Rube Goldbergian explanation of how the current state of the internet came to be?

> Who cares what the spelling is?

People trying to figure out whether an author means "tortious" or "tortuous."

I think it's very respectful to the author to make sure you're reading what they intended to write. When it comes to anything I write/say, please don't separate the art from the artist - just ask the artist what he meant to say.

(comment deleted)
Thanks to you I just learned the word tortious!
misspell one word... and it goes of on a tortuous tangent ;-)
The entry to my particular rabbit hole is to read any wikipedia article in math and follow the references up to the primary sources. Thanks to libgen and scihub nowadays you can access immediately the whole math corpus. You get to read the first proof of any theorem you fancy in a couple of clicks. We are living in a golden age for historians of math. This is awesome!

It feels really to be "exploring the internet". In a glorious quality and extent that I could have never imagined when my dad brought home a modem in 1995.

Counterpoint to all the gloom.

Most interesting content is indeed either

(a) on 5 or 6 sites, or linked from there (b) on professional news or media websites

This is because only a tiny proportion of people have the skills, or need, to build their own website with their own domain. 99% of people - including 99% of people with interesting things to say - will say them on Facebook, Twitter, Medium or Substack. Then there are people paid to be interesting. You'll find them on the NYT, or (for my region) the Eastern Daily Press.

This is fine! Web browsers are read-only. Certain websites, built on the web, provide services for writing. People use them.

yes, we just have to accept that those big aggregators won and the idea of truly distributed internet did not become a reality
This also is revealing - maybe it is time to create a blog after all.

Many people do give up on computer problems if they take longer than a few minutes to solve, or do not want to spend time piecing together fragments together, domain, hosting, dns, email, etc.

> Whatever happened to the idea of “exploring the Internet”?

Written content on the Internet are much lower in terms of quality than before. SEO has taken over and so written content tend to be wordy or salesy. There is no mechanism for users to flag low quality content to Search engines or other users, so it is a free for all.

Online forums are long gone. People migrated to Reddit where discussion is much more shallow, or private communities on Facebook.

We could still find interesting stuff on YouTube, for now.

If you want the experience again, the best way, as others have pointed out, is to search via sites and not crawlers - specifically community forums or non-crawled aggregators.

If you truly want to experience the frustration of the original, pre dominating search engine web, then I highly recommend tor sites. Many are not advertised in directories and the only way to find them is to find other interesting tor sites. Kind of a chicken and egg problem, but thankfully some basic search engines and dread exist to get the ball rolling.

(comment deleted)
It's still there. It's just not popular. I spent quite a few hours on IRC when Freenode imploded earlier this year.