> The technology for organizing the world’s knowledge should be owned by everyone.
This is what really nails it for me.
There's far too much black box in pretty much every major search engine out there. Maybe it's by design "so that people can't game it". Even so, it's not working very well.
I'm excited for the next 10 years to see what we (humans) come up with to solve the state of the internet, because something's gonna give at some point.
Why though? 99.9% of people don’t care, and the internet (Facebook, instagram, TikTok, YouTube, gmail, the occasional Wikipedia) work fine for them.
The overwhelming vast majority of people just don’t care about the internet outside those major few silos, so as far as “humanity” is concerned the internet is working as intended.
It pisses me off what’s become of the internet, but I don’t personally see it changing.
I guess it’s a good thing 99% of people don’t run the world. If products were dictated by the majority we wouldn’t have macOS, nor even iOS which is technically a minority by raw users. Obviously some users use more intensely than others, which is why catering to power users is a totally viable and legit strategy.
You are right about people's current opinions, but seem to be assuming that given a better option, people would continue to make sub-optomal choices.
If so, I don't agree. Your small minority's job is to deliver those alternatives, and to feed the flames while the rest of the world makes the transition. Which they will do because the next thing is clearly so much better.
Have more faith in most of humanity and doubt yourself and similar others for having failed so far to disrupt this industry with better technology.
I use duck duck go. Recently someone showed me their screen where they were using google to do a search. I was absolutely aghast. The last time I used google when you searched for something you saw a simple text list of sites (which is how DDG still works). Instead the google results were… a disaster. You had to scroll through some much garbage before finding actual search results - a list of sites. It was like google was saying, “here, look at all this trash instead of clicking a link and going to a different site”. When did google become so bad?
i have a guy in digital marketing tell me that his friend does SEO and he does wonders with obscure keywords and shit. that friend is a freelancer and earns a good payday.
When you want to insert your brand in every fucking imaginative keyword as opposed to people "searching for something",
why does internet advertising revolve around everyone assuming every person googling something "WANTS TO BUY SOMETHING"?
If every heavily SEO’d result was produced by a company that produced a directly relevant product, I don’t think we’d be as disappointed with the content.
The truely garbage content is produced as cheaply as possible (scraped, generated from a data source or generated via “ai”) to capture advertising revenue, often via sub prime advertising networks (or a number of middleman networks).
But to your point, not everyone wants to buy something, and not everyone needs to.
Much of the content out there is simply trying to capture your attention and make you available to some of the worst advertising and ad networks (read scams, lead gen, fake buttons, affiliate crap).
> why does internet advertising revolve around everyone assuming every person googling something "WANTS TO BUY SOMETHING"?
Rather I think it’s because everyone who buys ads has something to sell.
Search ads are “direct action”. You click a link to do something. Ads on eg. TV are more about “brand memory” - reminding you they exist. When you watch tv you’re passively taking in information, but when you’re searching you’re actively trying to click something already. It’s a better fit behaviorally.
Honestly that is probably safer. Having a typo in the url could Easley give you a phishing link. However, I also have gotten fishing adds when looking websites up so it’s not cut and dry at all.
For a good while if one searched for a (Dutch) gov institution or business google shows you the (free to call) phone number as a clickable link but the anchor has a different link to a paid per minute redirecting service. I know plenty of people who found the weird 15-50 euro entry on their bill.
Oh no, it's not. Google's ads have been used to do phishing a lot. And - at least a few years ago, it was extremely difficult to report such ads.
Perhaps it has improved recently, but it used to be a plague in crypto - people getting ads for phishing sites instead of legitimate ones, losing money, and Google being unresponsive to reports.
Honestly I think while this seems absurd if you're relatively knowledgeable about the internet, it's really not something that should be surprising or even particularly shameful? Like, why wouldn't they if it works, is easier to remember, and makes sense to them?
The reasons this is actually potentially bad are pretty deep in the internet wonk weeds, where you get into questions of gatekeepers and provinance of information and it shouldn't be surprising most people don't care about those things: those of us who do have failed to provide them with better tools.
On some level it's a little like saying "my dad sent me an email and he didn't use pgp! Can you believe it!??"
I know where DDG gets the results from, but in all the years using it, it has never failed to find what I am looking for. Or, I’ve never needed to check google because DDG didn’t find what I was looking for.
Yes? Offering ad space (and placing affiliate links) is how they make money.
Difference to Google is how they position themselves in regards to privacy, and that Google actually built a search engine. Both make their money by providing ad space.
Bing still refuses to index one of my pages, telling me to follow their rules. They won't tell me what rule I'm in violation of, though, and I can't tell that I'm in violation of any of them.
And this is educational content, text only, no ads or popups, no SEO hacking. Bing's analysis tool told me only that I was missing the "lang" attribute from my HTML tag. So I added it, but of course that wasn't the issue.
I reached out to them, and they replied saying that the page didn't meet the requirements for listing, but didn't elaborate.
It certainly makes me wonder what content their broken algorithm is missing.
And it sucks because it means DDG is missing that content, too.
No. It’s Bing in the same way that Uber is Elastic Search. It’s built on Bing and other tools and adds, tweaks, adjusts, etc. Calling DDG a Bing proxy is somewhere between misleading and dishonest.
Funny, I find them to be identical to Google's results localized for Sweden. Are you possibly using Google Search logged in or saving cookies between closing tabs (i.e. not using Cookie AutoDelete)?
Whoogle is best in class, but doesn't provide much benefit unless combined with a rotating VPN. It also doesn't solve the "GBY" problem, where the majority of search engines rely on Google/Bing/Yandex's indices instead of using their own.
This is especially dangerous because it propagates an illusion that there's dozens of engines to choose from. The reality is these three companies control more and more of humanity's ingress to information, censoring what they see fit for political/financial gain.
I do too. However, I am responsible for a couple of sites that Bing absolutely refuses to even index. Google has no issues with them, not for 10 years running. Those sites are effectively invisible in Bing and therefore DuckDuckGo. I’m not talking about low ranking, the entire domain is ignored.
My own personal website for example is not even listed in the search engine I use. Microsoft support see the issue but have no explanation and escalate and quote “quality requirements” for weeks now. A human review yielded positive results regarding quality. Meanwhile, I see lots of SEO spam on DDG when searching for generic technical terms.
Sure. A very simple one would be https://ipbl.herrbischoff.com, a public blocklist page referencing a resource used by a couple dozen users. The HTML doesn't get a lot simpler than that and is entirely valid markup. It's (unsurprisingly) low ranked in Google but it's there.
Not so in Bing, it's simply not there. The page exists since March 2021. Bing Webmaster Tools reads "Discovered but not crawled. URL cannot appear on Bing", giving no further reason. Also: "Last crawl attempted 01 Feb 2022 at 19:35", which means that Bing did not bother to retry for months, despite me submitting it manually on a regular basis. Clicking the "Live URL" tab results in entirely green checkmarks along with "URL can be indexed by Bing".
Another example would be my personal site: https://herrbischoff.com. Same issue. That one is listed on Google for more than 10 years.
This is fascinating, thanks. Have you experimented on allowing the bing ad bot that you have blocked? If they have some kind of retaliatory non-crawling?
Interesting theory. But the IPBL doesn’t even have a robots.txt and a different, larger site from a German celebrity I host does have the same directives and is indexed, although incompletely.
My working theory is that Bing’s selection algorithm is biased towards large and already popular sites. In the server logs, I don’t see Bing even attempting to crawl the sites I mentioned, except requesting robots.txt and the root page. Bing appears to be excruciatingly slow to update anything but high traffic sites.
Again, Microsoft Support was unable to explain this behavior even after manual, human review found everything to be in order.
I tried deleting robots.txt entirely and got only Chinese crawlers and SEO bots, but still no Bing crawl. All organic traffic comes from blogs linking directly and Google.
Bing constantly prioritizes the content to be indexed that will drive highest users satisfaction. Please follow Bing Webmaster Guidelines to better understand criteria for most valuable content.
I totally like the idea but I dare to doubt that this would solve the SEO problem. Website owners who are participating in those notorious affiliate programs or earn money with ads will still use the search engine to drag people onto their generic websites, using methodolgies to fit the search engines ranking mechanisms, no matter if they are public or not.
SEO and all it's results seem to be immanent to the system.
I'm hoping that having community moderated search results will limit this problem. The problem will be SEO people trying to infiltrate the community, which may be tricky to solve. But Wikipedia has had to grapple with similar problems, so I think it is solvable.
Sell weekly SEO guids to rank well and punish last weeks tips. (joking)
What you want to do is put the SEO monster in front of your cart and make it do useful work. You basically got an army of hard working people with money to burn who will do anything. What is there to complaint about?
You can have a semi-manual system, with user input. So websites filled with SEO get manually flagged by users. Of course, you need to trust your users as well... this can be done with a Web of Trust-style reputation system: users endorse each other, and you can build a reputation graph/tree that traces reputation and easily cull bad subtrees. If this endorsement system conserves reputation (you give a fraction of your reputation away each endorsement, and new reputation is never created), then it becomes sybil-proof, where it's not advantageous to create say millions of users to increase reputation.
why can't all the users that are being paid to promote spam endorse each other? And won't distributed trust systems just make some people "trust billionaires" and others "trust impoverished" though no fault of their own? If everyone trusts Oprah she'll have the types of same power people complain about billionaires people now. Basically influence. Also, anyone who's close to her gets the blessing of her influence where as if you're far removed you get nothing. Seems like the just reproducing the exact thing so many are trying to counter.
Mutual endorsement, if reputation is conservative, is like giving eachother reputation. If both users reputation is equal, there's no net gain (total reputation is always constant).
If one users reputation is higher, there's redistribution, but algorithms would need to carefully weight reputation linearly in all decisions, making redistribution not advantageous, solving any issues with sybil attacks[1].
Well, you need to define your objectives. No system is robust to a failure of all its actors. If every user (and even developer) is ill-intended, no system will give good results. So we need some "hopeful" (and accurate) assumptions.
One might be that the typical user can sensibly elect a few individuals to trust -- it could be developers (which are a natural choice for trust), to activist and publicly visible individuals (even close friends). Then presumably you could adopt his trust model (such individuals could be roots in independent conservative trust webs/graphs). I think a very large number of such webs might be computationally expensive, but hopefully you'd be able to find someone you trust or start your own independent graph (if you trust no one, you'd effectively lose all anti-SEO measures I guess). This very naturally leads to a decentralized reputation system!
He's doing that thing [1] where's he's writing about a thing and presumably wants me - the interested reader - to know more about that thing because it's the thing he's spending all his time on, but he gives zero navigational options to his thing. So as that interested reader, it's down to me to find the name of his thing [Mwmbl] and then (hilariously, given the context!) use a search engine (probably The Evil Google) to find HIS thing.
Seriously, people, if you're writing about anything at all, making assumptions is always a bad idea. If you're writing about a product, make it more than easy to get to it. Provide plentiful CTA's (that's Calls To Action, defined so as not to make the same mistake of assumption) - links, bittons, a big banner at the top: ("I'm building a non profit search engine called Mwmbl! Find out more").
IMO you also need an about page. Why are you doing this? Who are you? Why should the world pay attention? And stick them in your navigation. Overstate your cause rather than assume that people will get it. It's a rad product you're making! Don't undersell yourself! :-)
> The paid subscription model
> Donation funded, non-profit model
No! There is a 3rd! You could do a search app eco system where you leave the unlimited overly complicated puzzles a search engine could address as an exercise for the user.
I always have a bazillion ideas but couldn't think of a single good phone app before mobile phones. I mean, should I want my phone to be a gaming console? It seems ridiculous. Writing is writing books, all other kinds are watered down. Do I want to write books with an onscreen keyboard? It all sounded idiotic, nothing worth using.
But the idea you mention, typing an overly popular domain name without extension should take you to the website directly... What you are trying to say IMHO is CLI! Search is just the failback if the provided query/instruction doesn't make sense to any of the apps.
I cant think of many but there are no doubt thousands of activities that could benefit from an at least somewhat themed search engine. An app could be a biochemistry web directory that ranks results from a chosen sub folder above the normal results.
Any FOSS or other company could create a web dir tree with the few or many pages about it self. A check box lets you pick the ones you want to query. Normal results go under those results. The biochem wont bother you when searching for pokemon.
People love my stores. What they really want is to see illustrated results from my inventory above all other results. Uncheck the box if you are not in the mood. (edit: I'm joking of course but I do have a good fews shopping apps that I actually use)
Quite a neat way to crawl websites using a browser extension. That by itself is a form of donation to the search engine. Maybe in the future you can have dedicated software for self-hosted clients that users can run to crawl and index websites for mwmbl? Kinda like folding@home.
How are the batches of URLs to be crawled generated/discovered and posted at your API?
I have also thought that distributed crawling with the help of browser extensions, and/or clients like folding@home, could be a good idea. But how to deal with "spam injections"?
Maybe I misunderstand, but doesn't that mean you lose the benefit of having distributed crawlers if everything has to be crawled (again) locally somewhere?
YaCy can do distributed crawling and exchange the Indexes (in Peer to Peer mode). I have some node's who just receives and send indexes without crawling (much less storage intensive).
Get 3 people to scrape it and see if there are significant differences.
Some might, because of A/B testing or news updating, but even updating news will get a positive similar page and those that don't should probably fall into an exceptions category until it can be determined what can be done about it. Maybe a flag in the URL to give you a static page or just accept that it changes often enough that even faked pages won't last long?
Then I'll just add 3 million bots to the network (or just enough to have about 50%) and I can guarantee to win the A/B test against an honest client most of the time.
Then OP has to do things that don't scale: Review some pages and identify a subset that can be trusted. Then OP can compare their downloads to new accounts and mark the bots.
Then the botnet will just be honest for like a year before it abuses the network. Even better because now honest new clients can be kicked as they disagree with the bot majority. So now the network bleeds users.
Checking which account is honest isn't too hard, you detect that there is a "problematic mismatch" between two clients. So the project runs their own client to check. If one has an exact match, then you'd question the other.
There is a challenge for sites that serve different content based on GeoIP, A/B testing, dynamic content, etc. So some human review of the diff may help check for malice. If there's literally spam, human review would clearly detect this and that bot is distrusted.
Then I'll simply use more bots to get 80% of the network, then I can almost always win any disagreements and your "problematic mismatch" never triggers.
Plus I can now cause you to have to run your own crawler anyway and either slow progress or cost you a lot of money.
It's an arms race, but this is mostly a question of rate limiting account creation, assigning a trustworthiness score to different accounts, some network analysis to detect coordinated accounts, and having some trusted accounts (run by the project) that can help double check results. After an account loads poisoned data, you can detect this after the attack (user reported spam), and then block (or probably shadow ban) the malicious account.
You make it sound easy but companies have been trying to fight this stuff for ages.
You can buy a trustworthy residential IP for low cost, you can buy them in bulk in the thousands. All of them are real residential IPs from any ISP of your choosing in any country. You can rent Chrome browsers running over those IPs, directed via remote desktop and accessibility protocols (good luck banning that without running awful of anti-discrimination laws). You can do all that for under 1k$ a month for like 1 million clients.
My workplace has been at the other end of DDoS attacks directed by such services, best you can do is ban specific Chrome versions they use but that lasts until they update.
It's an uphill battle that you will loose in the long term if you rely on client trust.
In terms of spam injection (the concern from up thread) I don't think DDoS is relevant. If the core project manages asking clients to process URLs, they'd just IP ban any client that returns too many results. DDoS is a concern for other reasons though.
I think in this specific case, the spammer is on poor footing. The spammer wants to inject specific content, ideally many times. With double processing of URLs and the spammer controls 50% of the clients then there's a 50% chance that a simple diff would show the injected spam. The problem is that the spammer needs to do this many times, so their injection becomes statistically apparent. If the spammer can only inject a small number of messages before they are detected, then the cost per injected spam will be quite high. Long running spam campaigns could eventually be detected by content analysis, so the spammer also needs to rotate content.
Obviously you can play with the numbers, the attacker could try to control >>50% of the clients. The project could process URLs >2x. The project could re-process N% of URLs on trusted hardware, etc. It's not easy by any means, but you can tune the knobs to increase the cost for spammers.
> but this is mostly a question of rate limiting account creation, assigning a trustworthiness score to different accounts, some network analysis to detect coordinated accounts, and having some trusted accounts (run by the project) that can help double check results.
> Google tries to work out which sites are interesting by how long you spend on the site.
How would Google know how long you spend on the site? It only sees what links you clicked and doesn't know what happens next. (Unless the website uses Analytics, but Analytics doesn't affect search ranking.)
Let's not forget they also have a very popular browser that itself collects and bundles back home a lot of usage infomation from most of its user base:
I think this is a bit misleading. Chrome gets access to various data to enable various features -- for example to pass your location to the website (when you allow it to do so). Browsing history is used e.g. to power URL auto-completion.
This doesn't mean that this data can be used to inform Google Search ranking. That would be very shady and potentially illegal. I work for Google, and even though I do not work specifically on Search ranking, this doesn't sound like something that could be happening.
I'd be surprised if they don't have the capability to enable "linger time" statistics that collate frequency of new web site loads, memory demands, etc for performance.
This relates to the "how can they know how long I look at a web site for" asked above - if not specifically they do at least know the answer stochastically.
One feature that I really wish more search engines would have is the ability to blocklist certain domains, particularly ones whose results are never relevant or helpful to the query itself (Pinterest, Quora, etc). It could even be used as a factor in the site’s search rankings.
I think kagi does that, I use while was in beta, you also can assign a priority to sites, like normal or boost. Kagi is a paid service and doesn’t shows you ads.
I bet two-thirds of hn crowd is some how affiliated to the progress of search, ads, lead-generation, analytics, user-tracking (FAANG) etc. Think of their children...
As the underlying project discussed by this post is a search engine, I searched for “mwmbl” on mwmbl.org [0], and no results were found! Relevant results like the main site and GitHub repo show up when searched on Google or Kagi.
I searched for "Google" on mwmbl, and while the first page of results found many results, including Google Patents, Google Bug Hunters, and a Google Books page on a 2004 book about Anarchism, it did not find Google's home page.
I searched for "Elephant," and I got a Wikipedia page about a specific Elephant statue at Coney Island, a UK elephant charity, and a blog post about Haskell ("the elephant in the room").
It's unfair to poke fun at a very small project that admits that it is far from done yet, but it's gotta figure out a way to crack the "which pages are most likely to be relevant" problem or else it's not going to be useful.
To be honest, I didn’t entirely mean it as a comment on the search quality. But you would expect a search engine to return results about itself, and it was amusing it didn’t!
It does, I just tried it with 102.1. You can add a "keyword" to use a search engine, and once you've done that, you can add it as a search engine to the url+search bar (the main one on top, I don't know how it's actually called) as well, and you can set it as your default search engine.
I mean I used to have that but I seem to now. Did it move? This is 102.0.
Edit: You can make a bookmark and add a keyword, but that doesn't help me use it as a search engine. It used to be you could just create one from the contextual menu, what is this convoluted process.
First result is startrek.com, second result is Star Trek into Darkness IMDb but 3rd is xkcd.
It then goes off into Q and William Shatner Wikipedia links and Muppet Movie IMDB in Russian.
I tried putting a plus in front of IMDB and quoting Star Trek. It doesn't seem to be able to find Star Trek on IMDB. I admire the concept, and it is extremely fast.
I wouldn't be surprised if you don't want to talk about it because not could try and avoid the pitfalls, but what is your plan for avoiding bots trying to overwhelm the search scraper bot?
Are you looking to build a trusted network who can verify and validate other users responses on an undefined period?
> Instead of looking at how long people spend on a site, we would encourage users to give explicit feedback on rankings and use this to improve our ranking system.
While they're not wrong about how the way Google determines ranking has its issues, this way has its own set of problems. If you explicitly use user ratings as part of your rankings in some way, people can punish sites they don't like, ala review bombing on Yelp, Steam, etc.
Not saying it's necessarily a bad idea because of that, but I hope they don't fall victim to the mentality of, "let's just trust the users" as an ironclad rule, because that doesn't always work out well.
I went back to Reddit a week back after a 1 or so year, and what I saw was just disgusting on the main feed. One click, a bit of scroll and I was watching people calling republicans terrorists and being praised/upvoted/gifted. It's not just about bullshit at the top, it's also about echo chambers and the online equivalent of mob-behavior.
> it's also about echo chambers and the online equivalent of mob-behavior.
This particular example might be about the Republican party-wide support of echo chambers and offline mob behavior that led to an invasion of the building containing politicians certifying the vote of a newly elected leader and fueling disinformation about elections to weaken the trust and integrity of the system?
Regardless of this particular example, there has always been a pretty strong bias towards one side of US (and Australian, for that matter) partisan politics on the front page of Reddit; with over-the-top accusations towards the Republican Party as well as the LNP both receiving thousands of upvotes despite their poor quality.
Bombing is one thing, but you also have a whole SEO industry now that will exploit any way possible to get to the top of the rankings.
The moment you have community rankings on search, and your search gets popular, you land in a war zone with bots trying to mangle those. Reddit is kind of good dealing with that, but it is very resource intensive.
What if you limit accounts to real people and then keep track of their credibility? It's some initial effort but how could the ranking be manipulated when all dishonest people have burned their credibility?
You can’t limit to real people. If you managed to, I would make a service where people can sign up and I’ll pay to use their account. No cost to them and they make some money from it, seems totally reasonable.
That's just another group of people with reduced credibility. At some point, the price to incentivize the next person to offer their account is more expensive than the benefits of link manipulation. Every corrupted account can be discovered because the manipulated content stands out and will be reported.
I've always wondered if you could combat via referal-only sites. To get in you need to achir your humanness on someone else's account. If an account is found to invite too many spammers, robots, or otherwise it is banned or disallowed to invite more accounts.
I'm sure you could still manage to make "fake" accounts but it would be much more difficult, and linking them together would be much easier.
Of course starting a site like this would be very difficult. But maybe you could start without it then add it in once you get to a decent popularity such that many people can find a referral if they need to.
Of course it is a much smaller site so it isn't clear how effective this strategy would be at a large scale. Even a referral-based site approaching HN levels would be very interesting to see.
In China, all social accounts must be associated with a phone number, and phone numbers are tied to government identities. It doesn't stop any manipulation of scores and rankings.
> then keep track of their credibility?
It is very likely China will do that too soon. I think you can already imagine the ramifications.
This seems really really naive. Do you really think a non-profit is going to fight the hordes of spammers, scammers, seo masses, mechanical turk hordes, etc that are going to game your system?
Not sure I missed the sarcasm here.
Why would a non-profit not fight spam etc.? Why would a for-profit fight spam do so?
I see the largest internet companies, including Facebook, Twitter, and also Google, fight spam and other harmful content only to the degree absolutely necessary to stay somewhat usable. Which makes sense because it's costly and does not generate profit.
I would expect a non-profit, however, to focus much more on fighting harmful content because it centers around the user experience, hence quality of the content.
I don't see a guarantee this works in practice, but the respective incentives seem clear.
The general answer here is that a nonprofit cannot maintain the (massive) amount of resources it takes to address spam/abuse/blackhat SEO etc, while a for-profit entity ostensibly has a profit motive to do a decent job, and the resources to do so.
When a for profit entity is more successful at fighting abuse, their users are happier and they sell more ads and so can devote more resources to fighting spam. When a nonprofit successfully fights spam, they don't get more resources, and the spammers upgrade their toolboxes, because they do have a profit incentive.
Like @daoudc writes, Wikipedia shows that the nonprofit can maintain the resources because they attract volunteers.
If you create a search engine where users can report spam and get some form of karma for valid reports that is shown in their social network, then it's quite likely that the users have enough momentum to get ahead of the spam.
I don't know what the world needs but I need a personalized search engine. I would like to filter out anything to do with sports. I would like filter articles that contain marketing jargon and technobabble. I would like to filter articles written below high school grade level. And so on.
395 comments
[ 3.3 ms ] story [ 311 ms ] threadThis is what really nails it for me.
There's far too much black box in pretty much every major search engine out there. Maybe it's by design "so that people can't game it". Even so, it's not working very well.
I'm excited for the next 10 years to see what we (humans) come up with to solve the state of the internet, because something's gonna give at some point.
The overwhelming vast majority of people just don’t care about the internet outside those major few silos, so as far as “humanity” is concerned the internet is working as intended.
It pisses me off what’s become of the internet, but I don’t personally see it changing.
If so, I don't agree. Your small minority's job is to deliver those alternatives, and to feed the flames while the rest of the world makes the transition. Which they will do because the next thing is clearly so much better.
Have more faith in most of humanity and doubt yourself and similar others for having failed so far to disrupt this industry with better technology.
We (humans)? Are the fish up to something? Never did trust fish, especially regarding SEO stuff.
obviously good search trump's the dark mode tho.
i have a guy in digital marketing tell me that his friend does SEO and he does wonders with obscure keywords and shit. that friend is a freelancer and earns a good payday.
When you want to insert your brand in every fucking imaginative keyword as opposed to people "searching for something",
why does internet advertising revolve around everyone assuming every person googling something "WANTS TO BUY SOMETHING"?
The truely garbage content is produced as cheaply as possible (scraped, generated from a data source or generated via “ai”) to capture advertising revenue, often via sub prime advertising networks (or a number of middleman networks).
But to your point, not everyone wants to buy something, and not everyone needs to.
Much of the content out there is simply trying to capture your attention and make you available to some of the worst advertising and ad networks (read scams, lead gen, fake buttons, affiliate crap).
Because people googling SOMETHING are more likely to buy SOMETHING than people googling SOMETHING_ELSE.
Rather I think it’s because everyone who buys ads has something to sell.
Search ads are “direct action”. You click a link to do something. Ads on eg. TV are more about “brand memory” - reminding you they exist. When you watch tv you’re passively taking in information, but when you’re searching you’re actively trying to click something already. It’s a better fit behaviorally.
Instead of writing facebook.com in the URL-bar, they search for facebook and click the first link...
https://www.abc.net.au/news/2022-06-21/scammers-using-text-m...
https://www.channelnewsasia.com/singapore/scam-bank-google-a...
I think the only solution is education.
Perhaps it has improved recently, but it used to be a plague in crypto - people getting ads for phishing sites instead of legitimate ones, losing money, and Google being unresponsive to reports.
The reasons this is actually potentially bad are pretty deep in the internet wonk weeds, where you get into questions of gatekeepers and provinance of information and it shouldn't be surprising most people don't care about those things: those of us who do have failed to provide them with better tools.
On some level it's a little like saying "my dad sent me an email and he didn't use pgp! Can you believe it!??"
The results are fine for me in day to day usage and I find that Google will not provide me with better results if I cant find it on DDG.
Would it be better if they had their own indexer? Maybe...
You use a proxied Bing. If you want a proxied Google (what I prefer), you can use https://startpage.com
Difference to Google is how they position themselves in regards to privacy, and that Google actually built a search engine. Both make their money by providing ad space.
I did no such thing but looked around and found the company had been sold to an advertiser/tracker. It used to work much, much better than ddg:(
And this is educational content, text only, no ads or popups, no SEO hacking. Bing's analysis tool told me only that I was missing the "lang" attribute from my HTML tag. So I added it, but of course that wasn't the issue.
I reached out to them, and they replied saying that the page didn't meet the requirements for listing, but didn't elaborate.
It certainly makes me wonder what content their broken algorithm is missing.
And it sucks because it means DDG is missing that content, too.
https://github.com/benbusby/whoogle-search
This is especially dangerous because it propagates an illusion that there's dozens of engines to choose from. The reality is these three companies control more and more of humanity's ingress to information, censoring what they see fit for political/financial gain.
My own personal website for example is not even listed in the search engine I use. Microsoft support see the issue but have no explanation and escalate and quote “quality requirements” for weeks now. A human review yielded positive results regarding quality. Meanwhile, I see lots of SEO spam on DDG when searching for generic technical terms.
Not so in Bing, it's simply not there. The page exists since March 2021. Bing Webmaster Tools reads "Discovered but not crawled. URL cannot appear on Bing", giving no further reason. Also: "Last crawl attempted 01 Feb 2022 at 19:35", which means that Bing did not bother to retry for months, despite me submitting it manually on a regular basis. Clicking the "Live URL" tab results in entirely green checkmarks along with "URL can be indexed by Bing".
Another example would be my personal site: https://herrbischoff.com. Same issue. That one is listed on Google for more than 10 years.
My working theory is that Bing’s selection algorithm is biased towards large and already popular sites. In the server logs, I don’t see Bing even attempting to crawl the sites I mentioned, except requesting robots.txt and the root page. Bing appears to be excruciatingly slow to update anything but high traffic sites.
Again, Microsoft Support was unable to explain this behavior even after manual, human review found everything to be in order.
I tried deleting robots.txt entirely and got only Chinese crawlers and SEO bots, but still no Bing crawl. All organic traffic comes from blogs linking directly and Google.
—————
Thank you for your patience!
After further review, it appears that your site < http://herrbischoff.com https://ipbl.herrbischoff.com/> did not meet the standards set by Bing the last time it was crawled.
Bing constantly prioritizes the content to be indexed that will drive highest users satisfaction. Please follow Bing Webmaster Guidelines to better understand criteria for most valuable content.
>I found it very slow the last time I tried it
Maybe you used a node that was running on a rpi ;)
SEO and all it's results seem to be immanent to the system.
What you want to do is put the SEO monster in front of your cart and make it do useful work. You basically got an army of hard working people with money to burn who will do anything. What is there to complaint about?
(previous suggestion here: https://news.ycombinator.com/item?id=31585340)
If one users reputation is higher, there's redistribution, but algorithms would need to carefully weight reputation linearly in all decisions, making redistribution not advantageous, solving any issues with sybil attacks[1].
[1] https://en.wikipedia.org/wiki/Sybil_attack
One might be that the typical user can sensibly elect a few individuals to trust -- it could be developers (which are a natural choice for trust), to activist and publicly visible individuals (even close friends). Then presumably you could adopt his trust model (such individuals could be roots in independent conservative trust webs/graphs). I think a very large number of such webs might be computationally expensive, but hopefully you'd be able to find someone you trust or start your own independent graph (if you trust no one, you'd effectively lose all anti-SEO measures I guess). This very naturally leads to a decentralized reputation system!
I believe you mean 'inherent'.
immanent:
adjective
existing or operating within; inherent.
"the protection of liberties is immanent in constitutional arrangements"
synonyms: inherent, intrinsic, innate, built-in, latent, essential, fundamental, basic, ingrained, natural
Great, detect the affiliate links and downrank them. Ban it completely if the problem is severe or the products are fake/etc.
> earn money with ads
Similar, detect them and downrank. If a particular ad network is knowingly malicious, ban it entirely.
Seriously, people, if you're writing about anything at all, making assumptions is always a bad idea. If you're writing about a product, make it more than easy to get to it. Provide plentiful CTA's (that's Calls To Action, defined so as not to make the same mistake of assumption) - links, bittons, a big banner at the top: ("I'm building a non profit search engine called Mwmbl! Find out more").
K, thanks, </ moan >
[1] https://news.ycombinator.com/item?id=31494925
[1] https://github.com/mwmbl/mwmbl#how-do-you-pronounce-mwmbl
> The paid subscription model > Donation funded, non-profit model
No! There is a 3rd! You could do a search app eco system where you leave the unlimited overly complicated puzzles a search engine could address as an exercise for the user.
I always have a bazillion ideas but couldn't think of a single good phone app before mobile phones. I mean, should I want my phone to be a gaming console? It seems ridiculous. Writing is writing books, all other kinds are watered down. Do I want to write books with an onscreen keyboard? It all sounded idiotic, nothing worth using.
But the idea you mention, typing an overly popular domain name without extension should take you to the website directly... What you are trying to say IMHO is CLI! Search is just the failback if the provided query/instruction doesn't make sense to any of the apps.
I cant think of many but there are no doubt thousands of activities that could benefit from an at least somewhat themed search engine. An app could be a biochemistry web directory that ranks results from a chosen sub folder above the normal results.
Any FOSS or other company could create a web dir tree with the few or many pages about it self. A check box lets you pick the ones you want to query. Normal results go under those results. The biochem wont bother you when searching for pokemon.
People love my stores. What they really want is to see illustrated results from my inventory above all other results. Uncheck the box if you are not in the mood. (edit: I'm joking of course but I do have a good fews shopping apps that I actually use)
I kept forgetting about Kagi. I have a login for that.
Yep.com has a different model, I haven't read into it far enough to decide if they actually do as they say they'll do.
- Access your data for all websites - Monitor extension usage and manage themes
How are the batches of URLs to be crawled generated/discovered and posted at your API?
How do you deal with duplicate crawls?
And since you control what URLs need to be crawled, you protect yourself against rogue clients sending arbitrary URLs.
There certainly are a lot of elegant ways to reduce spam for this particular problem imo.
I'm not worried about the URLs, but the content of the URLs sent back.
Say the server tells a client to crawl a CNN article. The "hacked" client sends a fake CNN article back.
Some might, because of A/B testing or news updating, but even updating news will get a positive similar page and those that don't should probably fall into an exceptions category until it can be determined what can be done about it. Maybe a flag in the URL to give you a static page or just accept that it changes often enough that even faked pages won't last long?
There is a challenge for sites that serve different content based on GeoIP, A/B testing, dynamic content, etc. So some human review of the diff may help check for malice. If there's literally spam, human review would clearly detect this and that bot is distrusted.
Plus I can now cause you to have to run your own crawler anyway and either slow progress or cost you a lot of money.
You can buy a trustworthy residential IP for low cost, you can buy them in bulk in the thousands. All of them are real residential IPs from any ISP of your choosing in any country. You can rent Chrome browsers running over those IPs, directed via remote desktop and accessibility protocols (good luck banning that without running awful of anti-discrimination laws). You can do all that for under 1k$ a month for like 1 million clients.
My workplace has been at the other end of DDoS attacks directed by such services, best you can do is ban specific Chrome versions they use but that lasts until they update.
It's an uphill battle that you will loose in the long term if you rely on client trust.
I think in this specific case, the spammer is on poor footing. The spammer wants to inject specific content, ideally many times. With double processing of URLs and the spammer controls 50% of the clients then there's a 50% chance that a simple diff would show the injected spam. The problem is that the spammer needs to do this many times, so their injection becomes statistically apparent. If the spammer can only inject a small number of messages before they are detected, then the cost per injected spam will be quite high. Long running spam campaigns could eventually be detected by content analysis, so the spammer also needs to rotate content.
Obviously you can play with the numbers, the attacker could try to control >>50% of the clients. The project could process URLs >2x. The project could re-process N% of URLs on trusted hardware, etc. It's not easy by any means, but you can tune the knobs to increase the cost for spammers.
You make it sound easy. ;)
It changed a bit in the implementation.
How would Google know how long you spend on the site? It only sees what links you clicked and doesn't know what happens next. (Unless the website uses Analytics, but Analytics doesn't affect search ranking.)
https://www.ghacks.net/2021/03/16/wonder-about-the-data-goog...
This doesn't mean that this data can be used to inform Google Search ranking. That would be very shady and potentially illegal. I work for Google, and even though I do not work specifically on Search ranking, this doesn't sound like something that could be happening.
This relates to the "how can they know how long I look at a web site for" asked above - if not specifically they do at least know the answer stochastically.
They did that for a while without considering the obvious downside - the incentive to mass blacklist your competitors.
It's really useful!
Google’s name was so ubiquitous it became a verb, Duck Duck Go is a smart memorable name.
Mwmbl is a challenging product name, even if the .org domain name was available.
I was also confused by the name. That's not necessarily a good first impression.
[0] - https://mwmbl.org/?q=mwmbl
I searched for "Elephant," and I got a Wikipedia page about a specific Elephant statue at Coney Island, a UK elephant charity, and a blog post about Haskell ("the elephant in the room").
It's unfair to poke fun at a very small project that admits that it is far from done yet, but it's gotta figure out a way to crack the "which pages are most likely to be relevant" problem or else it's not going to be useful.
1. https://addons.mozilla.org/en-US/firefox/extensions/category...
Edit: You can make a bookmark and add a keyword, but that doesn't help me use it as a search engine. It used to be you could just create one from the contextual menu, what is this convoluted process.
1. Go to kagi.com
2. Right-click on the search field and choose "Add a keyword for this search..."
3. Fill in the keyword (e.g. kagi, as I did)
4. Right click in the top url+search field, and choose "Add Kagi search"
Now you can search via keyword, you can change to kagi while typing, and you can set the default search engine to kagi in the preferences.
Star trek imdb
First result is startrek.com, second result is Star Trek into Darkness IMDb but 3rd is xkcd.
It then goes off into Q and William Shatner Wikipedia links and Muppet Movie IMDB in Russian.
I tried putting a plus in front of IMDB and quoting Star Trek. It doesn't seem to be able to find Star Trek on IMDB. I admire the concept, and it is extremely fast.
I will help!
Are you looking to build a trusted network who can verify and validate other users responses on an undefined period?
While they're not wrong about how the way Google determines ranking has its issues, this way has its own set of problems. If you explicitly use user ratings as part of your rankings in some way, people can punish sites they don't like, ala review bombing on Yelp, Steam, etc.
Not saying it's necessarily a bad idea because of that, but I hope they don't fall victim to the mentality of, "let's just trust the users" as an ironclad rule, because that doesn't always work out well.
This particular example might be about the Republican party-wide support of echo chambers and offline mob behavior that led to an invasion of the building containing politicians certifying the vote of a newly elected leader and fueling disinformation about elections to weaken the trust and integrity of the system?
The moment you have community rankings on search, and your search gets popular, you land in a war zone with bots trying to mangle those. Reddit is kind of good dealing with that, but it is very resource intensive.
However it could just be reputation based like on Wikipedia
I'm sure you could still manage to make "fake" accounts but it would be much more difficult, and linking them together would be much easier.
Of course starting a site like this would be very difficult. But maybe you could start without it then add it in once you get to a decent popularity such that many people can find a referral if they need to.
In China, all social accounts must be associated with a phone number, and phone numbers are tied to government identities. It doesn't stop any manipulation of scores and rankings.
> then keep track of their credibility?
It is very likely China will do that too soon. I think you can already imagine the ramifications.
source: worked in ads and search for decades, incl google.
It's weird that more people don't know about it.
But also there should be more than one decentralized search engine.
I see the largest internet companies, including Facebook, Twitter, and also Google, fight spam and other harmful content only to the degree absolutely necessary to stay somewhat usable. Which makes sense because it's costly and does not generate profit.
I would expect a non-profit, however, to focus much more on fighting harmful content because it centers around the user experience, hence quality of the content.
I don't see a guarantee this works in practice, but the respective incentives seem clear.
When a for profit entity is more successful at fighting abuse, their users are happier and they sell more ads and so can devote more resources to fighting spam. When a nonprofit successfully fights spam, they don't get more resources, and the spammers upgrade their toolboxes, because they do have a profit incentive.
If you create a search engine where users can report spam and get some form of karma for valid reports that is shown in their social network, then it's quite likely that the users have enough momentum to get ahead of the spam.
There are very few incentives better than profit..This is going to be a US v/s USSR fight during the cold war..
Amen. How many times I searched for an astronaut's name - or any other person with significance - and instead I got results for some pro sports guy.
I guess 'significance' is subjective and I should search more specific.