Show HN: Search Engine for Blogs (blogsurf.io)
Blog discovery is a problem [0] due to the decentralized nature of online writing. Everyone writes on their own site or platform, and there’s no central place that brings everything together. Google results prioritize large media publications over blogs, so we need something else.
Blog Surf is an attempt to organize all of the great online writing done by individuals. I launched this project last year as a directory of personal blogs [1], but have now rebuilt it from scratch into a full-text search engine for blog posts.
You can search for blog posts, and filter by publish date and reading time. Blogs are manually reviewed before being added.
Posts are sorted by MarketRank [2], which is a measure of popularity across various online communities. Most projects that have attempted to organize blogs lack any way to measure the quality of a post, reducing their utility. With MarketRank, you can expect the top results for any query to be something you’d want to read.
The mental model for searching Blog Surf is “I want to see the best essays on X”
There’s also a directory so you can browse blogs by category, if you want a throwback to the Yahoo days.
If you’re a blogger yourself, you can check out the rankings page to see how your blog compares to others.
If you want to play around with things, we have a search API, and the full post dataset is also available for download.
[0] https://news.ycombinator.com/item?id=28591880
94 comments
[ 3.3 ms ] story [ 168 ms ] threadWould love to hear details about how you created the database, the infrastructure, etc if it's not a trade secret. Kudos on the launch!
Technorati was one of the inspirations here so that's great to hear.
> Would love to hear details about how you created the database, the infrastructure, etc if it's not a trade secret. Kudos on the launch!
Sure, it's actually fairly simple! The search backend itself is running on Typesense [0], which was very quick and easy to setup.
Due to the way ranking is calculated, I can actually avoid doing any real web crawling (though, I may add that in soon to help increase the index size). Ranking is based on submission to online communities, so all I really need is those submissions.
Using the Reddit, HN and Twitter APIs, I search for any submissions related to any blogs in the database, then those submissions give me the post URLs.
Once I have the post URLs, I just need to request those specific URLs to get the post data.
Then there's scripts for things like content extraction, inflation calculation, currency conversion etc.
All of those scripts are in python.
The frontend is a simple React app built with Next. All pages are statically generated.
Let me know if there's any more questions!
[0] https://typesense.org/
Any idea why?
I am working on increasing the amount of blogs significantly, but please bear with my modest index in the meantime.
My use case: often times blogs will respond to other blogs, linking to the original post in the process. The nature of linking means it's very easy to follow threads backwards in time, but given the original post it's often hard to discover the responses and ongoing conversation to follow things downstream. I'd like that downstream browsing to be easier.
My hope would be that such a tool could unlock higher-quality discourse. As a reader, this would let me hijack my natural tendency to follow comment threads, and redirect that attention towards slower-paced, more nuanced, more focused writing.
Edit: hmmm... though looking further, maybe this goes against your MarketRank philosophy.
> Edit: hmmm... though looking further, maybe this goes against your MarketRank philosophy.
It doesn't at all, but curious as to why you'd think that. We're not talking about using backlinks to rank pages after all, just as a discovery tool, which I think is great.
As to the source of my confusion... I think I was skimming and about to turn back to work and not thinking too deeply.
At the time I wrote that edit I had just gotten as far as "pagerank values links whereas marketrank values upvotes" (heavily paraphrased interpretation of 2.1 here: https://dkb.io/post/market-rank). My reaction was "oh, maybe that means links are bad... that's an interesting perspective but I don't have time to digest it right this moment. I'll put a pin in it and return to this article later to better understand, and hey I should add an edit to my comment to signal reading comprehension."
That last thought seems pretty ironic in retrospect.
But I don't feel like manual curation by one person is easily compatible with search engine. To me the content of your website is more suited to a weekly newsletter or something like that. Because after trying a few search "getting a job in vc", "best computer chair", "learning erlang" I'm not confident this answer better results than Google.
You've got a content size problem as you are manually curating, and this will lead to people not use your search as a default, and probably not use it as a search engine, but instead as a discovery system.
You can also try to get more blogs on your search engine, and create a community around it, if you want more more, you can follow this newsletter [1] and you will get probably 5 new blogs per day.
Congratz on the job, this is very cool
[1] https://hnblogs.substack.com
I see the manual curation as more of a temporary measure in the beginning. There are various ways blog detection can be automated and scaled, but manual curation for now gives me a better understanding of the data, and ensures I don't run into random edge cases.
> Because after trying a few search "getting a job in vc", "best computer chair", "learning erlang" I'm not confident this answer better results than Google.
Right now it's more useful for very broad queries like "inflation" or "covid". The index is pretty small at the moment, but the more posts that get added to the index, the more specific queries we'll be able to find good results for.
> You've got a content size problem as you are manually curating, and this will lead to people not use your search as a default, and probably not use it as a search engine, but instead as a discovery system.
That's actually what I want! This is not a search engine to replace Google, it's a discovery tool for blog posts.
Thanks for all the feedback here, and will definitely check out the newsletter.
https://blogsurf.io/about
Don't hate blogs and happy for resurgence, but repeating an uphill battle with indexing like it's 2007.
Also, random interesting posts on front page are like from 2009, 2011, 2015...... What? That's the freshest more relevant content?
Why would you expect a section titled "Random Interesting Posts" to have the freshest more relevant(?) content?
However, if you want fresher content, you can use the date range selector and set it to "Past Week" for the best posts of the week.
How do you figure out which are blogs and which arent'?
Tbh I'll probably use the random bit more than search but definitely going to keep checking back to pad my RSS feeds with interesting content.
That's interesting to hear, and fits well with the goals of the site. I want it to be more of a "discovery engine" than a "search engine". Search is one path to discovery, random posts are another, there are probably more.
One thing I'm thinking of adding is the ability to easily see the blog posts that any given post links to. If you see an interesting post, you could pull up everything that may be related.
> definitely going to keep checking back to pad my RSS feeds with interesting content.
Sadly not every blog has RSS, and many RSS feeds are incomplete. Another thing I would like to build is auto-generated RSS feeds for all blogs, which would also make it easy for people to programmatically parse any blog and do interesting things.
Curious - how do you know whether a site is a blog versus something else?
If you're submitting a blog and we don't already have a tag that fits, you can add it to the "Notes" section.
What I wanted were all his posts that weren't about Angular. So I tried adding -angular which works in Google. It pulled up one non-angular post and all the rest were the original ones that are there when you load the page. Add that one feature and I will probably use it a lot.
One little thing though: changing a search phrase or word and doing a new search, I notice the results do change, but there's no way to know if it really happened. Changing a Google search, the whole page flashes empty, that way I see/sense there's something new. In your case, a change is subtle, very subtle, too subtle. In one instance I had to look carefully to see the change in results.
Maybe adding a "you searched for X" is good enough, but I guess you can come up with a better way.
> changing a search phrase or word and doing a new search, I notice the results do change, but there's no way to know if it really happened. Changing a Google search, the whole page flashes empty, that way I see/sense there's something new. In your case, a change is subtle, very subtle, too subtle. In one instance I had to look carefully to see the change in results.
That is a good point. I did try to remove any loading indicators because I thought it would be smoother, but maybe it's a bit too smooth for people to realize their search went through. Will think more on how to fix this.
Popularity is certainly a flawed measurement, but it's hard to come up with a scalable way to determine quality that isn't flawed in some way.
Instead of being flawed by encouraging people to get tons of backlinks, this is flawed by encouraging people to do stuff that gets lots of upvotes.
Very open to more ideas on how to measure quality.
Are there any plans to check for redirects and update the URL or to recrawl?
Not sure what is a good way of creating a space for collaboration...
1. Exceptions are, e.g., ones that require two HTTP requests per query, such as Gigablast or ones that have strange limitations, e.g., Startpage, which has become unusable for me without Javascript. Contacting their "customer support" yielded no response.
Even better would be if search engines all shared their indexes and made these available for download. This would faciltate people building new search engines without needing to have their own index. In theory it would also bring a stop to the problem of people who submit large numbers of queries since all the bulk data they need would be available for download. www indexes that comprise public information could be freely shared as public data.
Doing it with APIs is a little tricky to make work in a usable way though. There have been various attempts at standardised APIs, e.g. OpenSearch[0], and metasearch engines like searX[1] have what are essentially pluggable scrapers, but there are still fundamental issues like getting different results at different times and having different ranking mechanisms.
Integrating at the index level could make a more usable search, but there are lots of other issues with this approach, e.g. those experienced with Apache Solr's Cross Data Centre Replication[2]. And yes, the volumes of data may also be an issue, given a search index will typically be slightly larger than the compressed data size, e.g. the 16M wikipedia docs are approx 32Gb compressed and approx 40.75Gb in a search index.
[0] https://github.com/dewitt/opensearch , unrelated to Amazon's Elasticsearch fork
[1] https://github.com/searx/searx
[2] https://solr.apache.org/guide/8_11/cross-data-center-replica...
I think most of us in this space are willing to collaborate so something interesting could happen.
[0] https://en.wikipedia.org/wiki/Metasearch_engine
[1] https://news.ycombinator.com/item?id=23771131
[1] https://developer.mozilla.org/en-US/docs/Web/OpenSearch