This. I'm glad these guys are running their own crawler, but their story doesn't add up.
Most likely this is an outfit that does a lot of webscraping and wants to minimize the amount of manual banning they incur. So they set up a plausible indie search engine as a front. Of course the search engine has to actually work when the scrapees notice and try it out.
sorry for the joke you can e.g. do a search for domain:*.com or domain:*.*
the index will be built up in the first step half a billion documents. we have already indexed more than 15 million domains, some with more than 10,000 pages from the domain. one million pdf documents are already available in the index and the good thing is that you can access all the results, unlike most search engines that only show you a few hundred results.
Cool to see a new general purpose search engine with its own crawler! I tried a couple of queries and it worked fine for me. Searching for for inetdex (https://www.tuxdex.com/?q=inetdex) returns 1,442 documents that apparently just reflect the crawler's user agent though. Might need some tuning. Can you (assuming OP is affiliated with the site) say something about the team and business model, if any? Would also be interesting to hear about what infrastructure you're running on and if there's any plans to open source the crawler.
I really like the minimal/90s look and feel too--keep up the good work, assuming it's not a honeypot! :^)
Yup, a plus from me about the clean 90's look, hope it will stay that way.
Searched for a term, gives +400k results, up to page 80 and still going. Another big plus from me since I just love to search low ranked results, sometimes you find true gems in obscure pages.
Our search server receives the request from the web server and returns the results. The web server, which could store the user IP, does not know the search query, since the POST method is used for transmission. The search server that processes the request does not know the user IP, only the web server.
The logs will be deleted after the visitor numbers have been determined.
Your search query remains anonymous at https://tuxdex.com.
I dont remember exactly, but it's possible to create a completely anon search engine. User encrypts the query using their key and sends it to search engine, search engine now doesn't know the query, but then it encrypts its entire database using public key of user and see if hashes meet for keywords?... something like that
I think you might be referring to homomorphic encryption. If I recall, the issues with it are the operations are expensive and it has a large overhead.
WRT the query not in the address bar - this is probably because then something like tuxdex.com/?terms="embarrassing search" doesn't show up in history or suggestions.
also to handle the case of browsers sharing the referrer url, though firefox only shares the domain and noreferrer html and security tags exist as well
The index is not full yet, our crawler is constantly working.
Today we indexed about 5000 documents from the https://apache.org domain, unfortunately this document is currently not in the index. But I can promise you that we will include at least 100,000 documents from apache.org.
Similar pages are not filtered[0]. Some queries return very weird results[1]. Also there is no way to add pages to be indexed. Overall 1000 times better than my private search engine, keep up the good work!
That's a feature I like, great for privacy and anonymity, relatively
easy to implement, and affords some resistance to bots (who will move
on to lower hanging fruit due to the connection set-up overhead)
Hope this new search engine will also offer an onion address.
I've been well served of late by SearX, specifically instances running the new-ish SearXNG (fork?).
SearX is a search aggregator, rather than having its own crawler, and as such a number of its instances getting temp suspensions for running too many requests in too short a time on the various search engines it queries. I still get good enough results from it.
I honestly don't pay that much attention, but they do have a decent list of potential search sources, so it's not something they're lazy about.
Additionally, this new search site (tuxdex.com) appears to just be a private/anonymous version of their existing search site (websearchengine.net) - although websearchengine.net isn't listed as a source for SearXNG either.
The search results come very fast, but there's some low hanging fruit QOL optimizations as far as I can see, mostly to do with search result ranking.
It doesn't seem to weigh the terms properly, there is some pull toward relevant terms but not as much as I'd expect from BM-25. I also think it doesn't promote results with adjacent terms enough. As a result the relevance of the search results sort of undulate up and down as you go through the results.
The de-duplication also seems non-existent. You can easily get a page full of duplicates of the same two links. You come a pretty long way if you just keep a simple hash of the HTML of each website or whatever.
This is made worse by what I think is a bug where the crawler follows redirects, and reports them as the original URL and not the redirected URL. I actually had the same bug in my crawler when I started out. Nasty anti-synergy with Wikipedia which has a lot of redirect.
These are relatively minor issues that shouldn't be hard to fix.
Thank you for the many posts and for visiting the TUXDEX.COM.
TUXDEX has received an update of both the software and the indexes, thanks bacon-and-stars we have been able to fix the error, TUXDEX is safe from XSS and other attacks.
About the results: We are working on our algorithm and other methods that will improve the results.
Please keep in mind that the well-known search engines sometimes employ thousands of people and have been on the market for years.
54 comments
[ 2.8 ms ] story [ 126 ms ] threadhttps://www.tuxdex.com/?q=query
Most likely this is an outfit that does a lot of webscraping and wants to minimize the amount of manual banning they incur. So they set up a plausible indie search engine as a front. Of course the search engine has to actually work when the scrapees notice and try it out.
I really like the minimal/90s look and feel too--keep up the good work, assuming it's not a honeypot! :^)
Searched for a term, gives +400k results, up to page 80 and still going. Another big plus from me since I just love to search low ranked results, sometimes you find true gems in obscure pages.
You could easily get 100 not requests for every 1 user request.
Problem solved.
The logs will be deleted after the visitor numbers have been determined. Your search query remains anonymous at https://tuxdex.com.
Does this just mean that it uses SSL/TLS?
> the query is not visible in the address bar.
Why would this be a privacy gain?
[1] https://private.sh/how-it-works.html
PIA is owned by Kape Technologies, https://en.wikipedia.org/wiki/Teddy_Sagi#Kape_Technologies_p...
Not as good as google at figuring out which terms are important.
weights API heavily, and doesn't return reasonable answers. For google, it's the first hit.For me this feels like a feature. Or just suggest. I don't want this to seacrh "corrected" keywords by default like Google does.
[0] https://www.tuxdex.com/?q=git
[1] https://www.tuxdex.com/?q=whale%20trades
For me personally, the mainstream independent search engine is Brave search, its genuinely good.
That's a feature I like, great for privacy and anonymity, relatively easy to implement, and affords some resistance to bots (who will move on to lower hanging fruit due to the connection set-up overhead)
Hope this new search engine will also offer an onion address.
the Goggles and Discussions features are great.
SearX is a search aggregator, rather than having its own crawler, and as such a number of its instances getting temp suspensions for running too many requests in too short a time on the various search engines it queries. I still get good enough results from it.
I was hoping to be able to use some esoteric ones, but that doesn't seem possible, unless I've overlooked something.
Being able to aggregate a couple of these up-and-coming engines with result reduplication sounds really neat.
Additionally, this new search site (tuxdex.com) appears to just be a private/anonymous version of their existing search site (websearchengine.net) - although websearchengine.net isn't listed as a source for SearXNG either.
But in terms of a real search engine, I find Brave to be the best.
If didn't use Searx, I would probably be using Brave.
the searx instance I use uses brave and google by default.
And the search engine is actually pretty decent (from my test).
The search results come very fast, but there's some low hanging fruit QOL optimizations as far as I can see, mostly to do with search result ranking.
It doesn't seem to weigh the terms properly, there is some pull toward relevant terms but not as much as I'd expect from BM-25. I also think it doesn't promote results with adjacent terms enough. As a result the relevance of the search results sort of undulate up and down as you go through the results.
The de-duplication also seems non-existent. You can easily get a page full of duplicates of the same two links. You come a pretty long way if you just keep a simple hash of the HTML of each website or whatever.
This is made worse by what I think is a bug where the crawler follows redirects, and reports them as the original URL and not the redirected URL. I actually had the same bug in my crawler when I started out. Nasty anti-synergy with Wikipedia which has a lot of redirect.
These are relatively minor issues that shouldn't be hard to fix.
https://tuxdex.com (Anonymous Search Engine)
What a weird thing to put as a feature.
It looks like the html tags from one of the search results messed up the page
TUXDEX has received an update of both the software and the indexes, thanks bacon-and-stars we have been able to fix the error, TUXDEX is safe from XSS and other attacks.
About the results: We are working on our algorithm and other methods that will improve the results. Please keep in mind that the well-known search engines sometimes employ thousands of people and have been on the market for years.
http://tuxdex.com (Anonymous Search Engine)