Some people like a flashy UI, the modern look is important for them. It's ok to have aesthetic preferences, let's not pretend we don't all have them.
In the end, it's a niche search engine I've made, the intended audience is the long tail. It just isn't for everyone, and if it was for everyone, it probably would be lesser for it.
I'm glad you aren't trying to please anyone. I'd like a return to an internet with fewer colors, gadgets and gizmos, custom fonts, TypeKit, JavaScript requirements, and so on. Most of the time I'm reading articles, so just give me more text and less fluff!
Ironic comment, considering that this is a search engine to weed out sites with awful UIs. This gives us exactly what we need in a search UI - no more, no less - in a clean and intuitive way.
Yeah the design could use some work. The search results are not compact - I only see 1 result without scrolling, not counting the related wikipedia link that apparently has no description.
I don't particularly like that it seems to be a column constrained to 550px width, instead of being responsive and taking advantage of greater widths.
to the author of the site, if you're not really into design/css, take a look at tailwindcss, it makes it fairly easy to produce a minimal amount of css that is responsive.
The page isn't prompting for cookie preferences, asking to allow notifications, popping up a mailing list or coupon half way do the page, playing a full page video with sound, or load 97million lines of javascript. I'd say its pretty much perfect.
Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)
I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke.
A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays.
So when you search for two words, it translates them into IDs using the special hash map, goes to the words file and finds the least common of the words; starts with that.
Then it goes to the words file and looks up the URL range of the first word.
Then it goes to the words file and looks up the URL range of the second word.
Then it goes through the less common word's range and does a binary search for each of those in the range of the more common word.
Then it grabs the first N results, and translates them into URLs (through mariadb); and that's your search result.
I'm skipping over a few steps, but that's the very crudest of outlines.
Good stuff. I've also been toying with doing some homegrown search engine indexing (as an exercise in scalable systems), and this is a fantastic result and great inspiration.
Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.
Well just crunching the numbers should indicate what is possible and what isn't.
For the moment I have just south of 20 million URLs indexed.
1 x 20 million bytes = 20 Mb.
10 x 20 million bytes = 200 Mb.
100 x 20 million bytes = 2 Gb.
1,000 x 20 million bytes = 20 Gb.
10,000 x 20 million bytes = 200 Gb.
100,000 x 20 million bytes = 2 Tb.
1,000,000 x 20 million bytes = 20 Tb.
This is still within what consumer hardware can deal with. It's getting expensive, but you don't need a datacenter to store 20 Tb worth of data.
How many bytes do you need, per document, for an index? Do you need 1 Mb of data to store index information about a page that, in terms of text alone, is perhaps 10 Kb?
> What crawler are you using and what kind of crawling speeds are you achieving?
Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper.
>How do you rank the results (is it based on content only) or you have external factors too?
I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query.
> What is your personal preferred search option of the 7 and why?
I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement.
> Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible.
And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing.
And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it?
And what are the barriers to use something like this? You say yourself that you are using Google most of the time. Is jumping to check results on this engine going to be too much friction for most uses?
Can something like this exist in isolation? What kind of value would it need to provide for users to remember using it en-masse as an additional/primary vertical search like they do for Amazon?
Just thinking out-loud as I am also interested in the space (through http://teclis.com).
I think in part it may just be because I'm not trying to found a start-up, and I'm not trying to get rich quick. If I were, I would have dealt with this very differently. My integrity and the integrity of this project is far more important than my bank balance. Not everyone feels that way, and I can respect that, but I do.
Ultimately I think running something like this for profit would create really unhealthy incentives to make my search engine worse. Any value it brings, right now, it brings because it isn't trying to cater to every taste and every use case.
I also hate the constant "don't forget to slap the like and subscribe buttons"-shout outs of modern social media, even though I'm aware they it is extremely efficient. If I went down that route, I would become part of the problem I'm trying to help cure. I do feel the sirens' call though, it's intoxicating getting this sort of praise and attention.
I want this to be a long-term project, not a some overnight cinderella story.
In the end, my search engine is never going to replace google. It isn't trying to, it's trying to complement it. It's barely able now, but hopefully I can make it much better in the months and years to come.
I agree with everything you say. The 'subscribe and like buttons' would not help your conversion with HN readers, on the contrary. Trying to run this for profit also would not help your conversion with this audience.
So given your setup is already ideal for 'conversions' for this population (low profile, high integrity, no BS) I was simply genuinely surprised that only 11 people converted given enormous visibility/interest this thread had. Hope that makes sense.
I think it simply takes time to build trust. The threshold to sending someone money is high. I probably wouldn't send someone money based on a proof of concept and lofty ambitions alone.
I'd absolutely consider sending someone money if they kept bringing something of value into my life. If I want more people to join the patreon, I'll just have to earn their trust and support.
I think it's good not to have to depend on financial compensation for every single thing in your life, if you can be comfortable or do well otherwise.
This allows quite a bit of its own kind of freedom even if maximum financial opportunity is not fully exploited. Perhaps even because you are not grasping for every dollar on the table at all times.
You can do things without having to know if they will pay off, and if it turns out big anyway you can make money as a byproduct of what you do rather than having pure financial pursuit be the root of every goal.
The day Google first appeared on the full internet it was excellent of course because it had no ads.
Plus another excellent feature was you would get the same search results no matter who or where you were for quite some period of calendar time.
If something new did appear it was likely to be one of the new sites that was popping up all the time and it was likely to be as worthwhile as its established associates on the front page.
You shouldn't need to crawl nearly as fast if you can compensate by treading more suitably where those have gone before.
I’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?
The trick is that the list of URLs for each word already is in the URLs file.
The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence.
Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8.
The first range corresponds to the URLs 1, 5, 7; and the second 2, 5, 8.
If you search for hello world, it will first pick a range, the range for "hello", let's say (1,5,7); and then do binary searches in the second range -- the range corresponding to "world" -- (2,5,8) to find the overlap.
This seems like it would be very slow, but since you can trivially find the size of the ranges, it's possible to always do them in an order of increasing range-sizes. 10 x log(100000) is a lot smaller than 100000 x log(10)
No, I actually count the n-grams as distinct words (up to 4-grams). The main limiter is for that is space, so I only extract "canned" n-grams from some tags.
I would first search for the bigram hello_world, that's an O(1) array lookup; as then documents merely containing the words hello and world (usually not a good search result), that's the algorithm I'm describing in the parent comment.
Interesting, in my database (http://root.rupy.se) I have one file per word that contains the ids (long) of the nodes (URLs), so to search many words together I have to go through the first file and one by one see if I find matches in the second.
How does the range binary search work, does it just prune out the overlaps, how efficient is it and how much data do you have in there for say "hello" and "world" f.ex?
I love this, and I love (many of) the results so far! What I can't find on the site is detail about what "too many modern web design features" means. Is it just penalizing sites with tons of JavaScript?
Javascript tags are penalized the hardest, but it also takes into consideration density of text per HTML. There's also some adjustments based on text length, which words occur in the page, etc.
Very cool project! How many websites do you have in your index? And how did you go about building it?
I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.
Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text.
I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.
If you're open to sharing your index I could make a classifier for you, I do this for a living.
It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.
Love it, kudos! This is great for developers and others who Just Need Answers and not shopping or entertainment.
If you're looking for feedback, both from a UI design and utility standpoint, you might consider "inlining" results from selected sites, e.g. Wikipedia, stacked change, etc. Having worked on search for a long time, inlining (onebox etc) is a big reason users choose Google, and that channelers fail to get traction. If you're Serious(tm), dog into the publisher structure formats and format those, create a test suite, etc.
A word of caution: if this takes off, as a business it's vulnerable to Google shifting its algorithms slightly to identify the segment of users+queries who prefer these results and give the same results to those queries.
If Google starts showing interesting text-heavy links instead of vapid listicles and storefronts, I have accomplished everything I ever could dream of.
But I've since expanded my websites, so now I think these play a decent role in later iterations, although they are virtually all of them pages I've found eating my own dogfood:
You may already be aware of this, but the page doesn't seem to be formatted correctly on mobile. The content shows in a single thin column in the middle.
This is awesome. I've been looking for a long time for a search engine that basically takes everything Google does and does the opposite. Thank you for doing this, I will definitely be bookmarking it.
Is there a way to suggest or add sites? I went looking for woodgears.ca and only got one result. I also think my personal blog would be a good candidate for being indexed here but I couldn't find any results for it.
It's very rare that I see a project on HN I can see myself using. This is one. Like others have said, the results can be a little rough. But they're rough in a way I think is much more manageable than the idiosynchrosies of more 'clever' search engines.
I think you need to approach it more like grep than google. It's a forgotten art, dealing with this type of dumb search engine.
Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.
That switch happened some years ago. I've been unlearning and relearning how to use google for what feels like at least three or four years now.
The main pain-point, though, is that a lot of long-tail searches you could've used to find different results in years past, now seem to funnel you to the same set of results based on your apparent intent. At least, it has felt that way -- I'm not entirely sure how the modern google algorithm works.
I realized this a few years ago when I observed my wife find things faster on Google than me.
I appreciate that it is easier for newcomers but I still hate it personally after years and especially that they cannot even avoid meddling with my queries even when I try to accept the new system and use the verbatim option.
So, you are re-implementing Altavista, Lycos and other old search engines.
They used the naive approach: you searched for "steak", and they would bring the pages which included the word "steak".
The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks.
Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.
> The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks.
I don't see a lot of people investing in SEO to boost their Marginalia results.
> Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.
Then people fooled Google into showing the White House as top result when searching for "a miserable failure".
At the moment marginalia's approach of sorting pages into quality buckets based on lack of JS seems to be working extremely well, but of course it will be gamed if it gets popular.
However, I'd rather want SEO-crafting to consider itself with minimizing JS, rather than spamming links into every comment field on every blog across the globe ;-)
> I think you need to approach it more like grep than google. It's a forgotten art
A search engine that accepted regex as the search parameter would be amazing.
I actually used this method as a field filter for a bunch of simple internal tools to search for info. Originally people were asking for individual search capabilities, but I didn't want it to become a giant project with me as the implementor of everyone's unique search capability feature request - so I just gave them regex, encoded inputs into the URL query string so they can save searches - gave em a bunch of examples to get going and now people are slowly learning regex and coming up with their own "new features" :P
But this made sense because it's a relatively small amount of data, so small that it's searched in the front end which is why it's more of a filter... I don't think pure regex would scale when used as a query on a massive DB, it would need some kind of hierachy still to only bother parsing a subset of relevant text... unless there is some clever functional regex caching algorithm that can be used.
Not OP but crawling is easy if you don't try scanning 5+ pages a second - almost all rate limiting/heuristic based 'keep server costs low' engines, including Cloudflare, don't care if you request every page, but will take action if you do something like burst every page and take up just as many server resources as a hundred concurrent users.
Now, that is assuming you aren't on some VPS provider. If you're going to crawl, you'll have the best chance when you use your own IPs on your own ASN, with DNS and reverse DNS set up correctly. This makes it so the IP reputation systems can detect you as a crawler but not one that hammers every site it visits.
Also, I imagine that, for a search engine like this, it doesn't expect content to change much anyways - so it can take its time crawling every site only once every month or two, instead of the multiple times a week (or day) search engines like Google have to for the constantly-updated content being churned out.
This is absolutely wonderful. I am LOVING the results I'm getting back from it: the sort of content-rich sites that have become nigh unreachable using traditional search engines. Thank you for building this!
I love this idea, and admire the work you put into it. I'm a fan of long reads and historical non-fiction, and Google's results are truly garbage.
I have a criticism that I think may pertain to the ranking methodology. I searched for "discovery of Australia". Among the top results were:
* A site claiming that the biblical flood was caused by Earth colliding with a comet (with several other pages from that site also making the top search results with other wild claims, e.g. that the Egyptians discovered Arizona);
* Another site claiming the first inhabitants of Australia were a lost tribe of Israel;
* A third site claiming that Australia was discovered and founded by members of a secret society of Rosicrucians who had infiltrated the Dutch East India Company and planned to build an Australian utopia...
These were all pages heavy with HTML4 tags and virtually devoid of Javascript, the kinds of pages you'd frequently see in the late 1990s from people who had built their own static websites in a text editor, or exported HTML from MS Word. At that time, there were millions of those sites with people paying for their own unique domain names, and so the proportion of them that were home to wild-eyed conspiracy theories was relatively small. What I think has happened is that kooks continued to keep these sites up - to the point where it's almost a visual trope now to see a red <h1> tag in Times New Roman and think, uh oh, I've stumbled on an "ancient aliens" site. Whereas scholars and journals offering higher quality information have moved to more modern platforms that rely more heavily on modern browsers - with or without their own domain names. So as a result what seemed to surface here were the fragments of the old web that remain live - possibly because people living in cabins in Montana forget to cancel their web hosting, or because the nature of old-school conspiracy theorists is to just keep packing their old sites with walls of text surrounded by <p> tags.
Arguably, this seems to rank the way Google's engine used to, since it couldn't run JS and they wanted to punish sites that used code to change markup at render time. At least, when I used to have to do onsite SEO work, it was always about simple tag hierarchies.
I wonder whether there isn't some better metric of validity and information quality than what markup is used. Some of the sites that surfaced further down could be considered interesting and valuable resources. I think not punishing simple wall-of-text content is a good thing. But to punish more complicated layouts may have the perverse effect of downranking higher-quality sources of information - i.e. people and organizations who can afford to build a decent website, or who care to migrate to a modern blogging platform.
Awesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so many websites for a project this size? Did you use any APIs like duck duck go or data from other search engines? Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?
> How are you able to keep the site running after HN kiss of death?
I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware.
My day job is also developing this sort of highly performance java applications, I guess it helps.
> What is your stack, elastic search or something simper?
It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries.
> How did you crawl so many websites for a project this size?
It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains).
> Did you use any APIs like duck duck go or data from other search engines?
Nope, it's all me.
> Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?
I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains.
It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.
This has amazing potential. I'd encourage you to form a non-profit, turn this into something that can last as an organization without becoming what you're trying to avoid becoming. This is a good enough start that I bet you could raise a sizeable startup fund very soon from a combination of crowdfunding and foundation grants—I bet the Sloan Foundation would love this!
Really like it, and love that you've done this yourself.
I'd prefer if it does just one thing and does that really well. Don't waste your time on calculator and conversion functions, or pseudo-natural language queries. There are plenty of good calculator & converter tools and websites, but we all need a really good search engine. I think you'd be better looking at handling synonyms and plurals.
I've mostly added these converters and frills because I'm trying to eat my own dogfood, and google is deeply engrained as my go-to source for unit conversions and back-of-envelope calculations.
Don't worry, this stuff is easy, and doesn't even remotely take away from the work on the harder problems.
It's custom java code, for the most part. I'm using mariadb for some ancillary information, but the index and the crawler and everything is written from scratch.
It's hosted on my consumer-equipment server (Ryzen 3900x, 128Gb ram, Optane 900p+a few IronWolf drives), bare bones on Debian.
I continue to want a communal set of 50-100 million urls and data that are "good" (i.e. for any value of good and more complete than common crawl) that are accessible enough to be easy to work with that can be used to experiment with different web search tech. There are enough separate problems to tackle in web search that breaking it down would maybe move the needle. We have lots of kaggle competitions about ranking, but using closed data. What other types of kaggle projects would help web search?
What can we do to foster a sustainable bazaar of projects to make it easier to build web search engines?
As everything in life flows in cycle, I predict the search engine that will de-throne Google will be like Google when it started - a simple variation of page rank.
No smarts, no bubble, no signals decided by over fitting to a biased engineer preference.
I wouldn't say the existence of this page proves your prediction right (as it's not dethroning Google anytime soon).
It's easy to forget that the goal of Google isn't to provide a useful search engine (at least not anymore), but the search engine is a by product of them wanting to show ads.
The search engine and the ads are tightly coupled. A better search engine means it can predict with more accuracy what you are looking for and can serve you an even more targeted ad that increases the chance you'll click.
Wouldn't the dethroner of Google be some new technology which is not a search engine like Google but better at solving the original task of finding information on how to solve problems?
Just like how iPad dethroned Windows PCs for average home user but not Mac because Windows had the monopoly and then an innovation destroyed MS in this space and not a competitor.
I don't think Google dethrones Yahoo and AltaVista scenario will occur again.
I do indeed index the web myself. Not the entire web, just a subset of it. The crawler quickly loses interest in javascript:y websites and only indexes at depth those websites that are simple. It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest (best-effort).
You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7 million URLs. I've gone as far as 50 million and could probably double that if I really wanted to. The difficulty isn't having a small enough index, but rather having a relevant enough index, weeding out the link farms and stuff that just take space.
I only index N-grams of up to 4 words, carefully chosen to be useful. The search engine, right now, is backed by a 317 Gb reverse index and a 5.2 Gb dictionary.
Not OP, but if I was to do this, I'd start by downloading Wikipedia and all its external links and references, and crawling from there. You should eventually reach most of the publicly visible internet.
I feel a little embarrassed that I didn't think of something like that.
When I did some crawler experimenting in my younger years, I thought I was pretty clever using sites that would let you perform a random Google searches. I would just crawl all the pages from the results returned.
Your method would undoubtedly be more interesting I think. It would certainly lead to interesting performance problems quicker, I bet.
I have only one recommendation that might make the search a bit more relevant, e.g when searching for 'linux locking' or 'kernel locking' kind of things.
Try to upsort things that match near the top of the content, like the top of the man page vs middle vs bottom.
One easy way to do it without having to store the positions, is to index the ngrams with max(sqrt,8) of their line number, this will cover first 64 lines, you can also use log() or just decide ad hock, top, middle, bottom of the document, so you can use only 3 values.
then just make the query "kernel locking" to "dismax(kernel_1 OR kernel_2 OR kernel_3...) AND dismax(locking_1 OR locking_2 ...) with some tiebreaker of 0.1 or so, you can also say "i want to upsort things on the same line, or few lines apart" by modifying the query a bit.
It works really well and costs very little in terms of space, i tried it at https://github.com/jackdoe/zr while searching all of stackoverfow/man pages and etc and was pretty surprised by the result.
This approach is a bit cheaper than storing the positions because positions are (lets say) 4 bytes per term per doc, while this approach has fixed uppre bound cost of 8*4 per document (assuming 4 byte document ids) plus some amortized cost for the terms
> It also focuses on websites in English, Swedish and Latin and tries to identify and ignore the rest
When I search for Japanese terms, it "says <query> needs to be a word", which wasn't the best error message. Maybe the error message should say something like "sorry, your language isn't support yet"?
Cool, I've been thinking on this topic a bit lately. Crawling is indeed not that hard of a problem. Google could do it 23 years ago. The web is a bit bigger now of course but it's not that bad. Those numbers are well within the range of a very modest search cluster (pick your favorite technology; it shouldn't be challenging for any of them). 10x or 1000x would not matter a lot for this. Although it would raise your cost a little.
The hard problem is indeed separating the good stuff from the bad stuff; or rather labeling the stuff such that you can tell the difference at query time. Page rank was nice back in the day; until people figured out how to game things. And now we have bot farms filling the web with nonsense to drive political agendas, create memes, or to drown out criticism. Page rank is still a useful ranking signal; just not by it self.
The one thing no search engine has yet figured out is reputability of sources. Content isn't anonymous mostly. It's produced and consumed by people. And those people have reputations. Bot content is bad because it comes from sources without a credible reputation. Reputations are built over time and people value having them. What if we could value people's appreciation relative to their reputability? That could filter out a lot of nonsense. A simple like button + a flag button combined with verified domain ownership (ssl certificates) could do the trick. You like a lot of content that other people disliked, your reputation goes down the drain. If you produce a lot of content that people like, your reputation goes up. If a lot of reputable people flag your content, your reputation tanks.
The hard part is keeping the system fair and balanced. And reputability is of course a subjective notion and there is a danger of creating recommendation bubbles, politicizing certain topics, or even creating alternative reality type bubbles. It's basically what's happening. But it's mostly powered by search engines and social media that actually completely ignore reputability.
> The hard part is keeping the system fair and balanced.
It is, which is why I think the author should stay away from anything requiring users to vote on things.
The problem with deriving reputability from votes over time is in distinguishing legitimate votes from malicious votes. Voting is something that doesn't just get gamed, it gets gamed as a service. You'll have companies selling votes, and handling all the busywork necessary to game the bad vote detector.
Search engines and social media companies don't ignore this topic - on the contrary, they live by it. The problem of reputation vote quality is isomorphic to the problem of ad click quality. The "vote" is a click event on an ad, and the profitability for both the advertiser and the ad network depend on being able to tell legitimate clicks and fake clicks apart. Ludicrous amounts of money went into solving this problem, and the end result is... surveillance state. All this deep tracking on the web, it doesn't exist just - or even primarily - to target ads. It exists to determine whether a real would-be customer is looking at an ad, or if it's a bot farm (including protein bot farm, aka. people employed to click on ads en masse).
We need something better. Something that isn't as easy to game, and where mitigations don't come with such a high price for the society.
What we need now is a search engine that weeds out sites that have been SEO optimized for keyword density.
I’m tired of searching for “generic keyword” and getting a page with an extremely low signal to noise ratio written like this:
“Many people search for generic keyword. That is why you can find all about generic keyword here. In fact we specialize in generic keyword and slight alterations of generic keyword.”
It’s like Google stopped caring that people were gaming it.
I love this! I've been searching random words with no aim in particular and keep finding lots of interesting tiny personal webpages. It feels like the old web
Yeah it's hosted by cloudflare. I'm currently IP-blocking them, as because they keep prompting my crawler with a captcha, presumably because it's made millions of requests from their CDN.
Some rigmarole getting recognized as a good bot by the CDNs. I've submitted a request fairly recently, but haven't heard back from them yet.
Like I would like to be on good terms with them, and other websites that block small independent crawlers.
I can't blame them though, there's a lot of bad bots out there. But I'm doing my best not be part of the problem.
Aha, I was going to ask how you were coping with CDNs like Cloudflare blocking bots. It's sad we've got to this point where basically only the established search engines are grandfathered in to be able to crawl sites.
I looked at the source for your site's front page. That's not text-heavy; that's markup-heavy. I didn't bother looking at the rest of the pages because it appears to be yet another crypto market site.
It's not always taking me to totally relevant sites but the results contain my favourite type of content.
Full of writing and pure html - usually the hallmark of someone who knows what they are doing, wants to communicate but doesn't want to waste their time.
The real test is in now, index server is reconstructing its index. It does this every 6 hours if there is new pages. Takes half an hour or so usually.
It's supposed to be able to handle searches at the same time, but jeepers, it's gonna have to chew through nearly 400 Gb of data while dealing with over 1 request per second.
Is your site/code on GitHub? I would be happy to give performance tips/tweaks. Also Fyi, https://marginalia.nu/ gives a certificate error (I know that's not the search site)
Coincidentally, the other day I was daydreaming about a search engine that favors sites that are updated less frequently. The thought being, the kinds of labors of love that characterized the 1990s Web that I still sometimes miss are still out there, it's just harder to find them amidst the flood of SEO dreck. So perhaps they could be made discoverable again with the help of a contrarian search engine that specifically looks for the kinds of things that Google and Bing don't like to see.
Million Short [1] offers an option to omit results from popular domains. It's a different approach from what you describe, but I think the goal is similar.
I think I like this method the most, honestly. There's something to be said for minimalist web design, but there's also something to be said for everything we've learned since the 90s.
Similarly, I wish there was a recommendation engine (for web, music, movies, whatever) that can show you what is the furthest away from your existing tastes. I've learned to re-create my Spotify account once every 6 months or so, as their recommendation engine becomes a boring machine after using it daily for some months.
I'd love to discover new content that is different from what I read/watch/listen to now, but it's really hard to know about genres you don't know about.
It's hard, though. I simultaneously want something far from my tastes, but I don't want to see Plandemic-style Ivermectin material, or Focus On The Family-style material. I want things that will push me out of my comfort zone sometimes, but it turns out I really don't want the thing furthest from my tastes; I want things marginally adjacent. I want them close enough to feel familiarity, but far enough that it challenges my worldview.
I don't think a recommendation engine can do that.
Doing that takes real work and curiosity. I'm afraid an algorithm will never be able to do it, particularly if you're into niche stuff. For instance I enjoy a lot a Japanese band called The Boredoms - but few people like it, and there's only 2 of their albums available in spotify.
I had this problem recently trying to fix an Atari. There's a guy out there who has ton's of guides on doing video out mods but newer guide references the older. However googling the OG guide didn't find it so I manually scoured his old web page.
It really depends on what you search for. A major drawback is that there needs to be text-heavy sites to find, in order for the search engine to find them.
Compare for example the results for "Duke Nukem 3D" with those for "Cyberpunk 2077".
Fascinating. I studied an "obscure" group of insects. My go-to search term to test an engine is their family name as it is a rarely used word and I know most (all?) of the major data sources that have accumulated data on it. When Wolfram Alpha added species names, I checked with the name, boring, Duck Duck, boring, Google (well we know Google isn't for search anymore, it's absolutely horrible) boring, Bing, boring... you get the idea.
This was a little different, extremely few results, but a couple of them really made me grin, and all(?) made me curious or raise an eyebrow or reflect on who/what might have been the source of the link, or remember some obscure connection from grad-school. So, if anything a crawled list of results worthy of ponder, thanks for this!
I'm intrigued by this experiment but I can't visualize it. What do you mean by boring results? Would combing through a library (the one with paper books) also produce boring results? What's your ideal results?
There are decades of academic research not digitized. The digitization window used to only hit around 1990, I haven't looked at it hard recently, but I suspect this still remains true for many important journals. This is grey only to those who do not know how to use a library.
Perhaps a counter example, something that is interesting. Anecdotally. This, of all things, is the top result in my search: https://tft.brainiac.com/archive/0303/msg00037.html. Which is strange to me because I don't recognize tft.brainiac. I click, it's a list of biological relationships among Hymenoptera, including a reference to genus of the wasps I studied, presumably in a biological relationship (host/parasite) context. I cataloged every relationship known at one point, so my brain wants to know where this come from, is it something I caught. Then I go look for more context, and find it's part of a thread about D&D(?) and hymenoptera, and it's epic, and a chunk of my morning is lost figuring out why and how this came to be.
If I understand it correctly, you're interested in bits and pieces of new information that's indirectly related to your object of interest. Degree 2 and 3 in Six Degrees of Kevin Bacon, so to speak. You know degree 0 like the back of your hand and you've seen almost everything closely connected. Finding novel, interesting things is getting more difficult.
Have you thought about cataloging all the related stuff you stumble upon? Something in between loose notes and what Moby Dick is to cetology.
> Have you thought about cataloging all the related stuff you stumble upon? Something in between loose notes and what Moby Dick is to cetology.
Tongue in cheek- new app time, to facilitate this. It should have the name "Degree4". Entries can only be made if degrees 2 and 3 are "defined". Scoffs at degrees 5 and 6, just because. Startup developing can probably unethically seed content by mining https://www.everything2.com/. Should use concepts of "AI" and "persistent homology"... profit!
But no, I don't outside a mental note. Closest I would come would be adding '!! <some note>' to my potwiki text notes (see my past comments) if its something I want to have come back with a grep, or think might be interesting to explore "when I retire". If it's a scientific fact in my field after researching it further it would go into this https://taxonworks.org (or its precursor).
In part, by boring results I mean I instantly recognize the top results, and I know exactly what will be in them, and I know which ones will actually contain potentially interesting new stuff, i.e. _I didn't have to search for these, I'd go their directly_. Then next results are all obscure, and I've already visited them, and/or I know they are historical and not something I have to revisit.
With this engine with at least 1/2 the links (to be fair there were < 20) I didn't recognize the URL at all, and it was clear in the text or the URL that there was an interesting bit to check out (i.e. what Google should have also returned after they barfed out the things I don't need to know about), but had never succinctly done in my experience.
I suppose the magic in this engine would have to be alerting the searcher that they found more of this type of link, as once I visited the 10 or so sites they would fall back into the "been there, done that" link category that Google appends somewhere after the ads and "big" sites, mixed in with a million search term spam sites, etc.
Not sure, but I remember when Google could find literally anything. Then they started adding a bunch of exceptions and crapped out their quality. I wonder how insanely different results would be to get the older Google Engine from the 2000s search result wise.
I now have to play games with Google to find things. I feel like I do less than I used to for some reason.
I was just thinking that they finally became Lycos. It's what all the search engines except Google looked like back in the early 2000s - ad laden cesspools of irrelevant search results and other content. And it's why we all switched to Google at the time.
You're absolutely correct, and a lot came from their nerfing of search modifiers like + - "search term" and whatnot. There's also a lot of ads and "PSA" type nonsense. If I'm looking for anything COVID related for example, I have to sift through a heap of PSA nonsense that's not even related to my search query.
Wacky idea: instead of Google changing it's algorithm every couple of years, it could run 50 algorithms in parallel leaving no way for sites to "optimise" for the current one.
"I haven't seen a black swan, ergo it's not real."
I've been burned by this decline in the past.
From creepy results i.e. first suggestion before typing was something I spoke near the Android and I never searched for before; to not finding what I was searching for before successfully, Google has started declining.
It would be nice if you provided a few real examples so that we would see how Google was so fantastic and found everything magically but then went to shit.
You want evidence? Search for a plumber/tradesperson in your area THEN try to find rational discourse about your options. There are literally 100s of results of websites remixing a small set of data, presenting it to you, and asking you to buy something to see more, when you know there is nothing behind the scenes.
This type of engine would punish these sites, in theory, and may turn up a discussion in some forum, newsgroup, etc. that is actually relevant, or insightful.
> Search for a plumber/tradesperson in your area THEN try to find rational discourse about your options.
I searched "plumber Austin TX" in Google and got a map and list of company websites near me. There are a lot of "top x y in z" list sites, but the top results were still the most relevant. I don't know what "rational discourse" I'm expected to find, though, or why I should assume the discourse I would find through Google is less rational than discourse I would find elsewhere.
I searched the same thing in OP and found nothing even remotely significant. Not even anything related to plumbing.
OP's project isn't optimized for relevance, it's optimized for nostalgia - providing a filter that keeps the modern web away and dropping quirky, interesting breadcrumbs to distract you and remind you of what it was like to wander around the web of the 90's.
Which is all well and good if that's what you want, and judging from the comments it is what a lot of people here want, but Google giving me a list of company names, numbers, websites and a map showing their location by distance is more useful, even if it uses "modern web design" and javascript.
> I searched "plumber Austin TX" in Google and got a map and list of company websites near me.
I think you could have done this historically in a Yellow Pages phone book. My OP used "boring". A list of plumbers is boring, been done on dead wood. I'm not saying boring != !useful.
> There are a lot of "top x y in z" list sites
This is an understatement. I actually want to know the top x in y, to do that I need "rational discourse". Rational discourse is recognizable as well written, insightful, humble, reflective, self-countering, anecdotal etc. By "search is terrible" I mean with respect to finding this.
> OP's project isn't optimized for relevance, it's optimized for nostalgia
Nostalgia is highly relevant if it's on topic, but agreeing with you as to what this engine is about.
>Rational discourse is recognizable as well written, insightful, humble, reflective, self-countering, anecdotal etc. By "search is terrible" I mean with respect to finding this.
I believe a search engine that ignores results based on superficial and aesthetic qualities like "modern web design" would be even worse in that regard, unless you're assuming no relevant discourse about any subject has taken place on the web since the early 2000's.
I admit, I have no idea what heuristic you would actually use to find "well written, insightful, humble, reflective, self-countering, anecdotal etc" content, but I've seen it on modern sites (even on Twitter,) and I've seen a lot of garbage on old sites, so a simple text search of only old websites doesn't seem like it.
> I believe a search engine that ignores results based on superficial and aesthetic qualities like "modern web design" would be even worse in that regard
I think you may be confused. No one's trying to replace Google here. The idea is to have another option for when Google craps out on you. And if you find that Google almost always craps out on you, then hey, maybe you're in the 5–10% who have just found their new default search engine :-)
well it displays a map of plumbers in my area, is it not useful?
Besides do you remember what it was displaying before "it became useless"? This whole thread is full of hand wavy claims with pretty much no good examples about how Google actually became worse in time. Hence my point.
Google Search is a fantastic product because it's essentially Spotlight for the web. It's by far the fastest way to get to things you already vaguely know are there and acts as a metasearch for large sites.
But as a result it's now less useful as a tool for scouring the web.
Mojeek was built in the same spirit (one server living in a house) and has 4.5 bn pages indexed now, and a bunch more servers. A lot of people comment in similar style of it reminding them of an older Internet, or generally less branded results. It's definitely an alternative point of view. Disclaimer: I work for them.
Likewise, I co-maintain the only "fan" site on one of my all-time favourite composers/performers, and gave the engine a shot with a unique string query. While my text-heavy WP-driven site didn't seem to make the cut, the results were highly relevant in that they were links to former band members and collaborators - a couple of which I didn't realize existed. That being said, there were a few sites (including my own) I expected to be returned, but no dice. Still, a fascinating experiment that many at HN have been clamouring for.
Exactly this. A couple results returned reference to obscure now-defunct newsletters and clubs, people that I know were historically important for past researchers, but only because this was my research forcus for so long would I have known this.
The search engine doesn't actually do full text search, so maybe your query was too... unique.
But do first of all verify that you haven't been hacked. There's about quarter of a million domains I've flagged that, besides their wordpress content, also host a ton of link spam crap off in some hidden folder. This reflects on the quality rating extremely negatively to the point where you may have not been indexed at all.
Secondly, are you behind cloudflare or some other big-name CDN? Because, as I mentioned in another comment, I can't crawl their pages without getting captchad until they approve of my humble request to be classified as a good bot.
There are some other hosting providers I flat out block on a subnet level because they host a large amount of link farms. This is currently Alibaba, Psychz, eSited, Cloud Yuqu and 1Blu.
Hmm, not sure what caused it to end up there, but I removed it from the blacklist. It still doesn't seem to want to index the domain however, probably CDN-related.
Thanks for the advice; not hacked, but I have "resurrected" many WP sites that have been (including my wife's non-profit). Just running on an EC2 micro instance, but I tried adding "site:" and received "No such domain". Actually, I think it's because I haven't enabled "HTTPS" yet! That's on my to-do along with migrating off EC2-Classic to VPC...
Is a good ol' .com with no ads and minimal JS - originally launched in 2011. Thanks again for your insights; I've bookmarked your site and will check back every so often to see if my site's been indexed.
If you're talking about the ads, I would bear in mind Google's whole business model is basically online advertising. Search is just the vehicle to deliver those ads; I'd say Google is pretty good at throwing things back.
Well, the unique value proposition is their gigantic index, really fast search and a bunch of other things.
I'm not sure about the quality of the results, I just use DuckDuckGo these days, but IMO the unique technical advancements are pretty unique to Google.
> If you search for "Plato", you might for example end up at the Canterbury Tales. Go looking for the Canterbury Tales, and you may stumble upon Neil Gaiman's blog.
I know it is just a suggestion, but had to try searching both, with no luck in getting the expected unexpected.
Yeah I did some work very recently aimed at improving the relevance a bit. It was a bit too random in the state it was before. Now it, perhaps, isn't random enough anymore.
Related question - suppose I want to create a meta search engine for myself, and I want it to be as fast as possible. What are the things I should be optimizing for?
I just looked up my last name and found a World class heavyweight weightlifter named Josef Grafl born in 1872 who has an awesome portrait of him on Wikipedia. Never before have I read about that man.
The website itself seems generated with some kind of kick-ass generator from template files (.gmi?)
I feel like I'm stuck with Wordpress.com because it brings me some traffic (whereas something hand-rolled on nsfspeech or digital ocean or whatever would literally be off the edge of the web), but the structure of that is so cool.
You can easily do proper SEO with static site generators too. Even more, static sites can be hosted via GitHub or GitLab Pages, Netlify or CloudFlare and in all cases the speed will outperform Wordpress in almost all cases. Also, you have way more control over the output than with Wordpress.
My pet peeve with search results is simply that there are ancient technical results that in many cases are irrelevant. If I am searching for a Window error message, I don't want some old forum post from 2001, especially if it didn't have any answers!
What would be cool would be for people who host old stuff to "archive" it at some point so it doesn't appear in normal results, only if you tick "include archives".
As much as the release names for macOS over the years were marketing gimmicks, it does make it a lot easier to zero in on the correct version when doing these types of searches.
Searched for my initials - got back a bunch of raw binary results (mp4, pdf, img, txz, etc.) which was disconcerting. Although it did find one reference to actual-me which is better than Google manages on the first 4 pages...
Yeah there was unfortunately a problem with the content-type code recently, it unfortunately categorized some binary data as HTML and tried to process it best-effort. So there's some binary soup in the index.
The bug has since been fixed, but it won't come into effect in a few weeks.
734 comments
[ 0.25 ms ] story [ 369 ms ] threadIn the end, it's a niche search engine I've made, the intended audience is the long tail. It just isn't for everyone, and if it was for everyone, it probably would be lesser for it.
I find it comfortable to use at 60% zoom level
it is litterally a search website with a text box for a search term and a button to do the search.
I don't particularly like that it seems to be a column constrained to 550px width, instead of being responsive and taking advantage of greater widths.
to the author of the site, if you're not really into design/css, take a look at tailwindcss, it makes it fairly easy to produce a minimal amount of css that is responsive.
A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays.
So when you search for two words, it translates them into IDs using the special hash map, goes to the words file and finds the least common of the words; starts with that.
Then it goes to the words file and looks up the URL range of the first word.
Then it goes to the words file and looks up the URL range of the second word.
Then it goes through the less common word's range and does a binary search for each of those in the range of the more common word.
Then it grabs the first N results, and translates them into URLs (through mariadb); and that's your search result.
I'm skipping over a few steps, but that's the very crudest of outlines.
Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.
For the moment I have just south of 20 million URLs indexed.
1 x 20 million bytes = 20 Mb.
10 x 20 million bytes = 200 Mb.
100 x 20 million bytes = 2 Gb.
1,000 x 20 million bytes = 20 Gb.
10,000 x 20 million bytes = 200 Gb.
100,000 x 20 million bytes = 2 Tb.
1,000,000 x 20 million bytes = 20 Tb.
This is still within what consumer hardware can deal with. It's getting expensive, but you don't need a datacenter to store 20 Tb worth of data.
How many bytes do you need, per document, for an index? Do you need 1 Mb of data to store index information about a page that, in terms of text alone, is perhaps 10 Kb?
How do you rank the results (is it based on content only) or you have external factors too?
What is your personal preferred search option of the 7 and why?
Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper.
>How do you rank the results (is it based on content only) or you have external factors too?
I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query.
> What is your personal preferred search option of the 7 and why?
I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement.
> Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible.
And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing.
I'm honestly a bit at a loss for words.
And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it?
And what are the barriers to use something like this? You say yourself that you are using Google most of the time. Is jumping to check results on this engine going to be too much friction for most uses?
Can something like this exist in isolation? What kind of value would it need to provide for users to remember using it en-masse as an additional/primary vertical search like they do for Amazon?
Just thinking out-loud as I am also interested in the space (through http://teclis.com).
Ultimately I think running something like this for profit would create really unhealthy incentives to make my search engine worse. Any value it brings, right now, it brings because it isn't trying to cater to every taste and every use case.
I also hate the constant "don't forget to slap the like and subscribe buttons"-shout outs of modern social media, even though I'm aware they it is extremely efficient. If I went down that route, I would become part of the problem I'm trying to help cure. I do feel the sirens' call though, it's intoxicating getting this sort of praise and attention.
I want this to be a long-term project, not a some overnight cinderella story.
In the end, my search engine is never going to replace google. It isn't trying to, it's trying to complement it. It's barely able now, but hopefully I can make it much better in the months and years to come.
So given your setup is already ideal for 'conversions' for this population (low profile, high integrity, no BS) I was simply genuinely surprised that only 11 people converted given enormous visibility/interest this thread had. Hope that makes sense.
I'd absolutely consider sending someone money if they kept bringing something of value into my life. If I want more people to join the patreon, I'll just have to earn their trust and support.
This allows quite a bit of its own kind of freedom even if maximum financial opportunity is not fully exploited. Perhaps even because you are not grasping for every dollar on the table at all times.
You can do things without having to know if they will pay off, and if it turns out big anyway you can make money as a byproduct of what you do rather than having pure financial pursuit be the root of every goal.
Plus another excellent feature was you would get the same search results no matter who or where you were for quite some period of calendar time.
If something new did appear it was likely to be one of the new sites that was popping up all the time and it was likely to be as worthwhile as its established associates on the front page.
You shouldn't need to crawl nearly as fast if you can compensate by treading more suitably where those have gone before.
The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence.
Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8.
The first range corresponds to the URLs 1, 5, 7; and the second 2, 5, 8.
If you search for hello world, it will first pick a range, the range for "hello", let's say (1,5,7); and then do binary searches in the second range -- the range corresponding to "world" -- (2,5,8) to find the overlap.
This seems like it would be very slow, but since you can trivially find the size of the ranges, it's possible to always do them in an order of increasing range-sizes. 10 x log(100000) is a lot smaller than 100000 x log(10)
Funny I also selected "hello" and "world" above! Xo
My system is also written in Java btw!
Here are example results of my word search:
http://root.rupy.se/node/data/word/four
http://root.rupy.se/node/data/word/only
etc.
But yeah, in short pseudocode:
I do use streams, but that is the bare essence of it.I would first search for the bigram hello_world, that's an O(1) array lookup; as then documents merely containing the words hello and world (usually not a good search result), that's the algorithm I'm describing in the parent comment.
How does the range binary search work, does it just prune out the overlaps, how efficient is it and how much data do you have in there for say "hello" and "world" f.ex?
Interesting idea. Definitely see an overlap with eReader markets and looking at text only contents.
How does it work?
It ignores pages on which it detects frameworks for ui and ads or any javascript code at all?
I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.
Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text.
I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.
If you're looking for feedback, both from a UI design and utility standpoint, you might consider "inlining" results from selected sites, e.g. Wikipedia, stacked change, etc. Having worked on search for a long time, inlining (onebox etc) is a big reason users choose Google, and that channelers fail to get traction. If you're Serious(tm), dog into the publisher structure formats and format those, create a test suite, etc.
A word of caution: if this takes off, as a business it's vulnerable to Google shifting its algorithms slightly to identify the segment of users+queries who prefer these results and give the same results to those queries.
Hope this helps!
https://xkcd.com/810/
I use external libraries for parsing HTML (JSoup) and robots.txt; but that's about it.
https://www.marginalia.nu/00-l%C3%A4nkar/
But I've since expanded my websites, so now I think these play a decent role in later iterations, although they are virtually all of them pages I've found eating my own dogfood:
https://memex.marginalia.nu/links/fragments-old-web.gmi
https://memex.marginalia.nu/links/bookmarks.gmi
You may already be aware of this, but the page doesn't seem to be formatted correctly on mobile. The content shows in a single thin column in the middle.
Is there a way to suggest or add sites? I went looking for woodgears.ca and only got one result. I also think my personal blog would be a good candidate for being indexed here but I couldn't find any results for it.
Unfortunately this doesn't seem to be a feature which new search engines are focusing on - Brave Search also misses that feature...
Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.
But just a week ago I found out that these "how", "what" questions give better and faster results on Google.
The main pain-point, though, is that a lot of long-tail searches you could've used to find different results in years past, now seem to funnel you to the same set of results based on your apparent intent. At least, it has felt that way -- I'm not entirely sure how the modern google algorithm works.
I appreciate that it is easier for newcomers but I still hate it personally after years and especially that they cannot even avoid meddling with my queries even when I try to accept the new system and use the verbatim option.
They used the naive approach: you searched for "steak", and they would bring the pages which included the word "steak".
The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks.
Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.
The rest is history.
I understand they are trying to maximize ad revenue and search does work very well for people who are looking for products or services.
But it no longer works well for finding information that is even slightly obscure.
I don't see a lot of people investing in SEO to boost their Marginalia results.
Then people fooled Google into showing the White House as top result when searching for "a miserable failure".
At the moment marginalia's approach of sorting pages into quality buckets based on lack of JS seems to be working extremely well, but of course it will be gamed if it gets popular.
However, I'd rather want SEO-crafting to consider itself with minimizing JS, rather than spamming links into every comment field on every blog across the globe ;-)
A search engine that accepted regex as the search parameter would be amazing.
I actually used this method as a field filter for a bunch of simple internal tools to search for info. Originally people were asking for individual search capabilities, but I didn't want it to become a giant project with me as the implementor of everyone's unique search capability feature request - so I just gave them regex, encoded inputs into the URL query string so they can save searches - gave em a bunch of examples to get going and now people are slowly learning regex and coming up with their own "new features" :P
But this made sense because it's a relatively small amount of data, so small that it's searched in the front end which is why it's more of a filter... I don't think pure regex would scale when used as a query on a massive DB, it would need some kind of hierachy still to only bother parsing a subset of relevant text... unless there is some clever functional regex caching algorithm that can be used.
Now, that is assuming you aren't on some VPS provider. If you're going to crawl, you'll have the best chance when you use your own IPs on your own ASN, with DNS and reverse DNS set up correctly. This makes it so the IP reputation systems can detect you as a crawler but not one that hammers every site it visits.
Also, I imagine that, for a search engine like this, it doesn't expect content to change much anyways - so it can take its time crawling every site only once every month or two, instead of the multiple times a week (or day) search engines like Google have to for the constantly-updated content being churned out.
I have a criticism that I think may pertain to the ranking methodology. I searched for "discovery of Australia". Among the top results were:
* A site claiming that the biblical flood was caused by Earth colliding with a comet (with several other pages from that site also making the top search results with other wild claims, e.g. that the Egyptians discovered Arizona);
* Another site claiming the first inhabitants of Australia were a lost tribe of Israel;
* A third site claiming that Australia was discovered and founded by members of a secret society of Rosicrucians who had infiltrated the Dutch East India Company and planned to build an Australian utopia...
These were all pages heavy with HTML4 tags and virtually devoid of Javascript, the kinds of pages you'd frequently see in the late 1990s from people who had built their own static websites in a text editor, or exported HTML from MS Word. At that time, there were millions of those sites with people paying for their own unique domain names, and so the proportion of them that were home to wild-eyed conspiracy theories was relatively small. What I think has happened is that kooks continued to keep these sites up - to the point where it's almost a visual trope now to see a red <h1> tag in Times New Roman and think, uh oh, I've stumbled on an "ancient aliens" site. Whereas scholars and journals offering higher quality information have moved to more modern platforms that rely more heavily on modern browsers - with or without their own domain names. So as a result what seemed to surface here were the fragments of the old web that remain live - possibly because people living in cabins in Montana forget to cancel their web hosting, or because the nature of old-school conspiracy theorists is to just keep packing their old sites with walls of text surrounded by <p> tags.
Arguably, this seems to rank the way Google's engine used to, since it couldn't run JS and they wanted to punish sites that used code to change markup at render time. At least, when I used to have to do onsite SEO work, it was always about simple tag hierarchies.
I wonder whether there isn't some better metric of validity and information quality than what markup is used. Some of the sites that surfaced further down could be considered interesting and valuable resources. I think not punishing simple wall-of-text content is a good thing. But to punish more complicated layouts may have the perverse effect of downranking higher-quality sources of information - i.e. people and organizations who can afford to build a decent website, or who care to migrate to a modern blogging platform.
I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware.
My day job is also developing this sort of highly performance java applications, I guess it helps.
> What is your stack, elastic search or something simper?
It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries.
> How did you crawl so many websites for a project this size?
It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains).
> Did you use any APIs like duck duck go or data from other search engines?
Nope, it's all me.
> Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?
I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains.
It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.
Is there any way that you can get an HTTP certificate?
I use an old iPhone 4S, and most of the modern web is inaccessible due to TLS. Hacker News and mbasic.facebook are two of the last sites I can use.
Usually text-based sites are more accessible, so this could be really useful to help me continue using my antique devices!
I'd prefer if it does just one thing and does that really well. Don't waste your time on calculator and conversion functions, or pseudo-natural language queries. There are plenty of good calculator & converter tools and websites, but we all need a really good search engine. I think you'd be better looking at handling synonyms and plurals.
Thanks.
Don't worry, this stuff is easy, and doesn't even remotely take away from the work on the harder problems.
It's hosted on my consumer-equipment server (Ryzen 3900x, 128Gb ram, Optane 900p+a few IronWolf drives), bare bones on Debian.
What can we do to foster a sustainable bazaar of projects to make it easier to build web search engines?
No smarts, no bubble, no signals decided by over fitting to a biased engineer preference.
It's easy to forget that the goal of Google isn't to provide a useful search engine (at least not anymore), but the search engine is a by product of them wanting to show ads.
The search engine and the ads are tightly coupled. A better search engine means it can predict with more accuracy what you are looking for and can serve you an even more targeted ad that increases the chance you'll click.
Or they'll continue to use it out of sheer inertia. Google is paying Apple $15 billion to keep its place as iOS default search engine.
IE6 didn't die overnight when Firefox arrived.
Example: `rust slow compilation site:stackoverflow.com`
Just like how iPad dethroned Windows PCs for average home user but not Mac because Windows had the monopoly and then an innovation destroyed MS in this space and not a competitor.
I don't think Google dethrones Yahoo and AltaVista scenario will occur again.
is this true? in the US, perhaps? because in south america it couldn't me more far away from truth - didn't happen at all
Or you can put in two ID's and have it find sources that both parties trust.
You'd be surprised how much you can do with modern hardware if you are scrappy. The current index is about 17.7 million URLs. I've gone as far as 50 million and could probably double that if I really wanted to. The difficulty isn't having a small enough index, but rather having a relevant enough index, weeding out the link farms and stuff that just take space.
I only index N-grams of up to 4 words, carefully chosen to be useful. The search engine, right now, is backed by a 317 Gb reverse index and a 5.2 Gb dictionary.
When I did some crawler experimenting in my younger years, I thought I was pretty clever using sites that would let you perform a random Google searches. I would just crawl all the pages from the results returned.
Your method would undoubtedly be more interesting I think. It would certainly lead to interesting performance problems quicker, I bet.
It's a directed search so it doesn't seem to need a particularly solid seed to get decent results.
I have only one recommendation that might make the search a bit more relevant, e.g when searching for 'linux locking' or 'kernel locking' kind of things.
Try to upsort things that match near the top of the content, like the top of the man page vs middle vs bottom.
One easy way to do it without having to store the positions, is to index the ngrams with max(sqrt,8) of their line number, this will cover first 64 lines, you can also use log() or just decide ad hock, top, middle, bottom of the document, so you can use only 3 values.
e.g. https://www.kernel.org/doc/html/v5.0/kernel-hacking/locking.... would do unreliable_1 guide_1 locking_1 ... then at line 4 kernel_2 locking_2 ... after line 50 ... then_7 ... and after that everything will be _8.
then just make the query "kernel locking" to "dismax(kernel_1 OR kernel_2 OR kernel_3...) AND dismax(locking_1 OR locking_2 ...) with some tiebreaker of 0.1 or so, you can also say "i want to upsort things on the same line, or few lines apart" by modifying the query a bit.
It works really well and costs very little in terms of space, i tried it at https://github.com/jackdoe/zr while searching all of stackoverfow/man pages and etc and was pretty surprised by the result.
This approach is a bit cheaper than storing the positions because positions are (lets say) 4 bytes per term per doc, while this approach has fixed uppre bound cost of 8*4 per document (assuming 4 byte document ids) plus some amortized cost for the terms
When I search for Japanese terms, it "says <query> needs to be a word", which wasn't the best error message. Maybe the error message should say something like "sorry, your language isn't support yet"?
You see stuff like this sometimes, makes me a bit sad.
https://linux.die.net/robots.txt
The hard problem is indeed separating the good stuff from the bad stuff; or rather labeling the stuff such that you can tell the difference at query time. Page rank was nice back in the day; until people figured out how to game things. And now we have bot farms filling the web with nonsense to drive political agendas, create memes, or to drown out criticism. Page rank is still a useful ranking signal; just not by it self.
The one thing no search engine has yet figured out is reputability of sources. Content isn't anonymous mostly. It's produced and consumed by people. And those people have reputations. Bot content is bad because it comes from sources without a credible reputation. Reputations are built over time and people value having them. What if we could value people's appreciation relative to their reputability? That could filter out a lot of nonsense. A simple like button + a flag button combined with verified domain ownership (ssl certificates) could do the trick. You like a lot of content that other people disliked, your reputation goes down the drain. If you produce a lot of content that people like, your reputation goes up. If a lot of reputable people flag your content, your reputation tanks.
The hard part is keeping the system fair and balanced. And reputability is of course a subjective notion and there is a danger of creating recommendation bubbles, politicizing certain topics, or even creating alternative reality type bubbles. It's basically what's happening. But it's mostly powered by search engines and social media that actually completely ignore reputability.
It is, which is why I think the author should stay away from anything requiring users to vote on things.
The problem with deriving reputability from votes over time is in distinguishing legitimate votes from malicious votes. Voting is something that doesn't just get gamed, it gets gamed as a service. You'll have companies selling votes, and handling all the busywork necessary to game the bad vote detector.
Search engines and social media companies don't ignore this topic - on the contrary, they live by it. The problem of reputation vote quality is isomorphic to the problem of ad click quality. The "vote" is a click event on an ad, and the profitability for both the advertiser and the ad network depend on being able to tell legitimate clicks and fake clicks apart. Ludicrous amounts of money went into solving this problem, and the end result is... surveillance state. All this deep tracking on the web, it doesn't exist just - or even primarily - to target ads. It exists to determine whether a real would-be customer is looking at an ad, or if it's a bot farm (including protein bot farm, aka. people employed to click on ads en masse).
We need something better. Something that isn't as easy to game, and where mitigations don't come with such a high price for the society.
I’m tired of searching for “generic keyword” and getting a page with an extremely low signal to noise ratio written like this:
“Many people search for generic keyword. That is why you can find all about generic keyword here. In fact we specialize in generic keyword and slight alterations of generic keyword.”
It’s like Google stopped caring that people were gaming it.
But I don't get good results for "rug pull".
- 1 https://rugpullindex.com
Some rigmarole getting recognized as a good bot by the CDNs. I've submitted a request fairly recently, but haven't heard back from them yet.
Like I would like to be on good terms with them, and other websites that block small independent crawlers.
I can't blame them though, there's a lot of bad bots out there. But I'm doing my best not be part of the problem.
I looked at the source for your site's front page. That's not text-heavy; that's markup-heavy. I didn't bother looking at the rest of the pages because it appears to be yet another crypto market site.
Now hoping for search engine that favors text-heavy sites and punishes paywalls
It's not always taking me to totally relevant sites but the results contain my favourite type of content.
Full of writing and pure html - usually the hallmark of someone who knows what they are doing, wants to communicate but doesn't want to waste their time.
It's supposed to be able to handle searches at the same time, but jeepers, it's gonna have to chew through nearly 400 Gb of data while dealing with over 1 request per second.
Coincidentally, the other day I was daydreaming about a search engine that favors sites that are updated less frequently. The thought being, the kinds of labors of love that characterized the 1990s Web that I still sometimes miss are still out there, it's just harder to find them amidst the flood of SEO dreck. So perhaps they could be made discoverable again with the help of a contrarian search engine that specifically looks for the kinds of things that Google and Bing don't like to see.
[1] https://millionshort.com/
I'd love to discover new content that is different from what I read/watch/listen to now, but it's really hard to know about genres you don't know about.
I don't think a recommendation engine can do that.
Clicking 'Surprise me' gave me an interesting article from 1994 http://milk.com/wall-o-shame/bucket.html
Compare for example the results for "Duke Nukem 3D" with those for "Cyberpunk 2077".
This was a little different, extremely few results, but a couple of them really made me grin, and all(?) made me curious or raise an eyebrow or reflect on who/what might have been the source of the link, or remember some obscure connection from grad-school. So, if anything a crawled list of results worthy of ponder, thanks for this!
If I understand it correctly, you're interested in bits and pieces of new information that's indirectly related to your object of interest. Degree 2 and 3 in Six Degrees of Kevin Bacon, so to speak. You know degree 0 like the back of your hand and you've seen almost everything closely connected. Finding novel, interesting things is getting more difficult.
Have you thought about cataloging all the related stuff you stumble upon? Something in between loose notes and what Moby Dick is to cetology.
> Have you thought about cataloging all the related stuff you stumble upon? Something in between loose notes and what Moby Dick is to cetology.
Tongue in cheek- new app time, to facilitate this. It should have the name "Degree4". Entries can only be made if degrees 2 and 3 are "defined". Scoffs at degrees 5 and 6, just because. Startup developing can probably unethically seed content by mining https://www.everything2.com/. Should use concepts of "AI" and "persistent homology"... profit!
But no, I don't outside a mental note. Closest I would come would be adding '!! <some note>' to my potwiki text notes (see my past comments) if its something I want to have come back with a grep, or think might be interesting to explore "when I retire". If it's a scientific fact in my field after researching it further it would go into this https://taxonworks.org (or its precursor).
With this engine with at least 1/2 the links (to be fair there were < 20) I didn't recognize the URL at all, and it was clear in the text or the URL that there was an interesting bit to check out (i.e. what Google should have also returned after they barfed out the things I don't need to know about), but had never succinctly done in my experience.
I suppose the magic in this engine would have to be alerting the searcher that they found more of this type of link, as once I visited the 10 or so sites they would fall back into the "been there, done that" link category that Google appends somewhere after the ads and "big" sites, mixed in with a million search term spam sites, etc.
Do you suggest anything better? As far as I can tell, all the other search engines are either repackaged Bing (ie: DuckDuck), or are just as bad.
I now have to play games with Google to find things. I feel like I do less than I used to for some reason.
The actual search results were another 1/4 page of completely identical results, followed by google ad placed search results.
I thought to myself, they've finally done it. Real responses are no longer first page.
A lot of the cause for google getting crappy, is "ok google", another "all platforms are the same" form of sickness.
No, a desktop is not a phone. No, voice searching is not the same as phone, or desktop.
Moores law means a modern day 2007-style Google should be significantly less expensive to run now than back then.
Also the most relevant patents are now free to use.
2021 Google is a sad story compared to 2007 Google and I'd actually pay to get back 2007 Google - ads included - meaning a double revenue source :-)
I've been burned by this decline in the past.
From creepy results i.e. first suggestion before typing was something I spoke near the Android and I never searched for before; to not finding what I was searching for before successfully, Google has started declining.
This type of engine would punish these sites, in theory, and may turn up a discussion in some forum, newsgroup, etc. that is actually relevant, or insightful.
I searched "plumber Austin TX" in Google and got a map and list of company websites near me. There are a lot of "top x y in z" list sites, but the top results were still the most relevant. I don't know what "rational discourse" I'm expected to find, though, or why I should assume the discourse I would find through Google is less rational than discourse I would find elsewhere.
I searched the same thing in OP and found nothing even remotely significant. Not even anything related to plumbing.
OP's project isn't optimized for relevance, it's optimized for nostalgia - providing a filter that keeps the modern web away and dropping quirky, interesting breadcrumbs to distract you and remind you of what it was like to wander around the web of the 90's.
Which is all well and good if that's what you want, and judging from the comments it is what a lot of people here want, but Google giving me a list of company names, numbers, websites and a map showing their location by distance is more useful, even if it uses "modern web design" and javascript.
I think you could have done this historically in a Yellow Pages phone book. My OP used "boring". A list of plumbers is boring, been done on dead wood. I'm not saying boring != !useful.
> There are a lot of "top x y in z" list sites
This is an understatement. I actually want to know the top x in y, to do that I need "rational discourse". Rational discourse is recognizable as well written, insightful, humble, reflective, self-countering, anecdotal etc. By "search is terrible" I mean with respect to finding this.
> OP's project isn't optimized for relevance, it's optimized for nostalgia
Nostalgia is highly relevant if it's on topic, but agreeing with you as to what this engine is about.
I believe a search engine that ignores results based on superficial and aesthetic qualities like "modern web design" would be even worse in that regard, unless you're assuming no relevant discourse about any subject has taken place on the web since the early 2000's.
I admit, I have no idea what heuristic you would actually use to find "well written, insightful, humble, reflective, self-countering, anecdotal etc" content, but I've seen it on modern sites (even on Twitter,) and I've seen a lot of garbage on old sites, so a simple text search of only old websites doesn't seem like it.
It is fun, though.
I think you may be confused. No one's trying to replace Google here. The idea is to have another option for when Google craps out on you. And if you find that Google almost always craps out on you, then hey, maybe you're in the 5–10% who have just found their new default search engine :-)
But as a result it's now less useful as a tool for scouring the web.
Broad and longer Twitter lists maintained here: https://twitter.com/SearchEngineMap/lists
But do first of all verify that you haven't been hacked. There's about quarter of a million domains I've flagged that, besides their wordpress content, also host a ton of link spam crap off in some hidden folder. This reflects on the quality rating extremely negatively to the point where you may have not been indexed at all.
Secondly, are you behind cloudflare or some other big-name CDN? Because, as I mentioned in another comment, I can't crawl their pages without getting captchad until they approve of my humble request to be classified as a good bot.
There are some other hosting providers I flat out block on a subnet level because they host a large amount of link farms. This is currently Alibaba, Psychz, eSited, Cloud Yuqu and 1Blu.
> Blacklisted false
> site:www.wsj.com
> Blacklisted false
> site:www.rt.com
> Blacklisted false
> site:www.nytimes.com
> Blacklisted true
?
If you're getting no such domain, it's either blocked because it looks too much like a spam domain, or it simply hasn't been discovered yet.
What's the TLD? I severely restrict some cheaper TLDs because they gave so much spam.
For example, cr.yp.to is an example of a baby I know I've definitely thrown out with the bathwater.
If you're talking about the ads, I would bear in mind Google's whole business model is basically online advertising. Search is just the vehicle to deliver those ads; I'd say Google is pretty good at throwing things back.
I'm not sure about the quality of the results, I just use DuckDuckGo these days, but IMO the unique technical advancements are pretty unique to Google.
Then w.r.t. quality of results, you're just using Bing: https://help.duckduckgo.com/duckduckgo-help-pages/results/so...
> If you search for "Plato", you might for example end up at the Canterbury Tales. Go looking for the Canterbury Tales, and you may stumble upon Neil Gaiman's blog.
I know it is just a suggestion, but had to try searching both, with no luck in getting the expected unexpected.
btw, interesting how many http (as opposed to https) sites show up...
I love this.
I feel like I'm stuck with Wordpress.com because it brings me some traffic (whereas something hand-rolled on nsfspeech or digital ocean or whatever would literally be off the edge of the web), but the structure of that is so cool.
What would be cool would be for people who host old stuff to "archive" it at some point so it doesn't appear in normal results, only if you tick "include archives".
It’s rudimentary but no IT-related result.
https://imgur.com/a/n2xro2Y
The bug has since been fixed, but it won't come into effect in a few weeks.