Lucene is the best search technology, in my humble opinion, and is amazingly fast. MongoDB has it built in (full-text search, based on Lucene) and I'm using it in the project I'm working on.
All platforms after a certain size need to be regulated as utilities.
When thousands or hundreds of thousands of companies live and die by your platform, its economic suicide to leave it in the hands of one single private entity.
A few things need to happen in order to guarantee fair play:
- Transparency: Your process can no longer be secret. People need to see whats going on to ensure no foul play is going on.
- The owners of the platform can't participate economically in it. Amazon is the obvious example. You don't want to be kicked out of your own platform? Don't open it to others. Its a simple matter of conflict of interest.
- DUE PROCESS: You can't have some minimum wage employee following a flowchart in call center making life of death decisions for companies living in your platform. There needs to be a proper and accessible appeals process with independent arbitrage if necessary. And above all, innocent till proven guilty. You can't fucking shut down my company and THEN I can go through the lengthy appeal. No company can survive that.
Too expensive? Tough shit, go wipe your tears with your trillion dollar market cap.
I'm sure there's a ton more that needs to happen. These are just the ones I can think about.
I don't buy this. Search placement gets exploited all the time. Hiding it hasn't worked. Ever. All history is against your argument.
Maybe it gets exploited because its made exploitable by design. How do you know if you can't check? You worry about bots but you don't worry about outright massive cheating from the company itself?
Yes, it would be worse. If Google ranked its own results higher (which I don't believe they need to do) then it's not that big a deal; I can just skip past them.
If every black-hat SEO knows how to cheat and they grab the first ten pages of results then the useful stuff gets lost in the noise and the search engine is no longer useful to me (or anyone else).
Even evil-Google wouldn't do that because they do have incentive to keep people coming back.
How do you know that doesn't already happen? Maybe instead of researching the algo and investing in manipulation, they just hand a check to someone at google and you're done.
Without transparency, you have no way of knowing.
Good old fashioned corruption is way more effective than modern hacker approaches.
Most of those points sound reasonable but where do you draw the line and wouldn’t these companies be able to just throttle your dependence on them right before that line?
Let’s take Otterbox and Amazon for example. If Amazon notices that Otterbox is selling exceptionally well, they can require proof that Otterbox is selling similar numbers on other platforms otherwise they will be throttled. Just like that, there’s no dependence on Amazon because you’re obligated to diversify your sales. Don’t like it? Tough shit, now you’re not allowed to sell on Amazon at all because you must maintain compliance with the law.
Also, while we’re labeling internet properties as utilities, can we label internet access a utility first?
Due process: a lot of this comes from regulation targeted at large tech cos. Eg my illegal site could be criminally prosecuted and shut down with a court order but the courts are too busy so they just hold the provider criminally liable instead, forcing the provider into violating due process.
They're not violating due process. There IS no process.
They just click a button and you're gone.
This needs to be regulated. A simple analogue of the civil process would suffice. Innocent till proven guilty would go A LONG way. These companies ban you at the first sign of trouble. They are very often wrong.
Yes, this puts a lot of burden on them. Like I said, tough titties. This is your prize for growing to such a behemoth size. You can't have your cake and eat it.
You're being a little myopic. This applies in all industries. Content creation is only one example.
You can have the best car in the world, but if all cars are bought in carmazon.com and they ban you, you're fucked. Sure, you can make your own outlet but you just lost billions in investment and you wont recover from it. All because someone clicked a checkbox in an amazon monkey moderation center.
Also, you're showing a ridiculous bias against content creators. They're a completely legitimate and respectable profession.
It's not the cost. I don't want shitty search, shitty stores, and crappy products just because some guy sent his friend $20 for "ISIS money" and then got banned from Venmo so he decided to whine on Twitter.
A lot of people, creators, merchants, etc; often get taken out of business by simple misunderstandings or arbitrary decisions from these platforms. Nobody is going to ruin anything, just have some form of due process where everyone's right to exist is respected.
If you release the search engine weighting process, you open up to gaming it. This is inherently an adversarial space. You will ruin it by asking Google to make the engine open.
For Google’s current product, yes. What if they would start to offer a paid ‘utility’ search that is both transparent and regulated, and part of the service fee would go to fighting adversaries?
Public search engine is not an isolated, academic problem to solve - it's a never-ending war against large economic interests that want to undermine it. This dynamic has huge implications.
People imagine Google without ads will be better or paid search makes it better - they are mistaken, since the moment Google (or any top search engines) completely stops serving any ads (or equivalent - everyone using ad block that blocks search ads), all of the companies spending money on ads will instead spend those money on SEO against those search engines. i.e. Selling ads is an effective way to limit the amount of money poured into SEO. As long as the cost of ads is lower than the amount you need to spend on SEO to get the similar amount of traffic through organic results, companies would rather buy ads. This adds more incentive for Google and other search companies to focus on fighting against SEO manipulations also, since a successful SEO effectively lowers the ad price companies are willing to pay. Paid search would make this problem actually worse - people who can afford to pay correlate strongly with the target demographic companies want to show ads, and that means once people starts signing up for paid search, companies will throw more money at SEO.
Now, the doomsday scenario (of SEO winning and all organic results becoming bad) may happen anyway because SEOs end up winning regardless despite Google and other search engines fighting against it. For humanity's sake, we better hope that's not how it ends up. It's less likely given some of the structural advantage search engines have, but SEOs have advantages too, so at this point it's not at all clear one side will necessarily win or lose.
The world economy runs on Google/Microsoft/Apple/Amazon. Imagine if any one of them were to fail, it would be catastrophic.
That said I'm not sure I trust the current American political climate to step in and regulate, and I'm not sure they have a right to... The influence of these businesses extends far outside the borders of the United States.
Adtech is a pretty big bubble. Failure would be disruptive. But from a long-term perspective I'm not convinced adtech is desirable or needed. Let's go to Luna and the icy moons instead.
As long as people who have next to nothing (a large portion of the world) can access valuable resources for free in exchange for looking at advertising, it is quite necessary. Even those who do have disposable income in first world countries are not interested in paying for a subscription to resources they rely on a daily basis. Ads are a necessary evil.
These regulations already exist and come up all the time.
US and EU have different payment process requirements. Same for data collection. There are numerous examples and all platforms find a way to implement them and comply.
Having some creator and merchant protections is not a technical issue. The US could legislate to protect US users and they could comply only in the US. No big deal.
Can't you be content with the fact that your work impacts and improves the lives of billions of people? What gives you the right to maintain a stranglehold over a market and drive off competition simply because you were lucky enough to be first to implement an idea, or had the fortune to have enough resources to stick it through the hard times?
Also note that companies are far from honest in this work - just look at what Google is doing to erode the web platform, and how it is leveraging it's search monopoly to drive businesses towards paying for ads.
I said regulation. You keep your company, but after a certain size, you're obliged to treat the OTHER hard working companies living on your platform with respect and not completely destroy them on a whim.
An example: A friend of mine worked for a battery selling business. They had most of their sales in Amazon. Everything was fine, they had a few years at this and around 100 employees.
One day, Amazon decided one of their papers or whatever wasn't good and closed them down. The appeal process too 3 weeks and bankrupted them.
Amazon was wrong.
Now, you think that's fair? All that hard work and all those people out on the street because some dumbass in amazon's underpaid customer service centers fucked up?
Like I'm saying, there needs to be due process. They shouldn't be able to that.
If you were dumb enough to run a business that only had a 3 week runway with ~100 employees and you relied entirely on someone else's platform without developing a sustaining channel for several years, that's not Amazon's problem - Amazon isn't responsible for other people's idiocy.
I agree that government often has to step in when companies reach a certain size.
The problem here is that government doesn't understand technology well enough.
Which makes sense, the majority of people don't. Your post is an example, no disrespect meant.
You can't have transparency in search. It's an arms race between search engines and black hat SEO gaming. It's been that way for decades, likely always will be. That's why the algos absolutely have to be secret, and there has to be no barrier to rapid iteration and innovation. The minute that changes, black hat wins and search goes back to looking like Altavista before it died. In that scenario everyone loses.
It's also imperative that there's a financial incentive to continue to innovate and stay (sort of) ahead in the arms race. It's easy to imagine that a government (or a nonprofit) could run search well, but that's only without an understanding of the scope and depth of the technology involved.
I'm not sure what the best solution is, but there's no question transparency isn't it.
I do, by the way, appreciate your idea of mandating more reasonable due process. However I'd add that if your business relies on search, you're gambling. It's often a smart gamble, for as long as it lasts organic rankings are free money, but be honest with yourself about it.
The competition is around serving the most relevant monetizable information. Google’s search quality for static content is arguably decreasing, while that kind of Search is a service I would personally like to pay for.
If Google were to “split up”, I as a customer would like a paid-for, transparant, utility search on one hand. My money can go to dealing with SEO abuse. On the other hand as a business I would like an ad-serving product, to which customers can voluntarily subject themselves to.
The two organizations would have to become separate business units, because I don’t want what’s currently happening: I search for information and get disinformation that others paid for.
Color me interested! There’s so many layers to measure: from natural language understanding to pagerank-style relevance ordering, filtering duplicates to presenting the different permanences of content (breaking news vs books on the same topic).
Yep. It's interesting. How do you deal with versioning. The same document 10 times on a CMS might overpower the search results. Does uniqueness rank higher? There you need a human judge at which search engine / search engine defaults is better.
> natural language understanding
I'd like to know if any search engines do this. I feel like it's out of scope. If they use the words "natural language understanding" I think they are running a scam and mean statistics.
I'd check misspelling. American vs British. Plurals. Non Dictionary words. Small words. Names. Initials. Exact match. So, quantifiable stuff.
Then prescriptive.
The idea would be a framework so someone can make (or grab their current data) the test different engines for different search terms against the data and mark which is correct or better.
You don't have to 100% agree what search result is better. But to just get to the stage where you can discuss that would be a great start.
And I suspect not agreeing what is better is not as contentious as one thinks.
I also think Google does search real well. So use that for a set point of the truth.
33 comments
[ 5.3 ms ] story [ 81.4 ms ] threadWhen thousands or hundreds of thousands of companies live and die by your platform, its economic suicide to leave it in the hands of one single private entity.
A few things need to happen in order to guarantee fair play:
- Transparency: Your process can no longer be secret. People need to see whats going on to ensure no foul play is going on.
- The owners of the platform can't participate economically in it. Amazon is the obvious example. You don't want to be kicked out of your own platform? Don't open it to others. Its a simple matter of conflict of interest.
- DUE PROCESS: You can't have some minimum wage employee following a flowchart in call center making life of death decisions for companies living in your platform. There needs to be a proper and accessible appeals process with independent arbitrage if necessary. And above all, innocent till proven guilty. You can't fucking shut down my company and THEN I can go through the lengthy appeal. No company can survive that.
Too expensive? Tough shit, go wipe your tears with your trillion dollar market cap.
I'm sure there's a ton more that needs to happen. These are just the ones I can think about.
Maybe it gets exploited because its made exploitable by design. How do you know if you can't check? You worry about bots but you don't worry about outright massive cheating from the company itself?
If every black-hat SEO knows how to cheat and they grab the first ten pages of results then the useful stuff gets lost in the noise and the search engine is no longer useful to me (or anyone else).
Even evil-Google wouldn't do that because they do have incentive to keep people coming back.
Without transparency, you have no way of knowing.
Good old fashioned corruption is way more effective than modern hacker approaches.
Let’s take Otterbox and Amazon for example. If Amazon notices that Otterbox is selling exceptionally well, they can require proof that Otterbox is selling similar numbers on other platforms otherwise they will be throttled. Just like that, there’s no dependence on Amazon because you’re obligated to diversify your sales. Don’t like it? Tough shit, now you’re not allowed to sell on Amazon at all because you must maintain compliance with the law.
Also, while we’re labeling internet properties as utilities, can we label internet access a utility first?
People seem to think the first guess is upon the House: https://news.ycombinator.com/item?id=22079174
They just click a button and you're gone.
This needs to be regulated. A simple analogue of the civil process would suffice. Innocent till proven guilty would go A LONG way. These companies ban you at the first sign of trouble. They are very often wrong.
Yes, this puts a lot of burden on them. Like I said, tough titties. This is your prize for growing to such a behemoth size. You can't have your cake and eat it.
I care way more about engines of innovation than whingey content creator types.
You can have the best car in the world, but if all cars are bought in carmazon.com and they ban you, you're fucked. Sure, you can make your own outlet but you just lost billions in investment and you wont recover from it. All because someone clicked a checkbox in an amazon monkey moderation center.
Also, you're showing a ridiculous bias against content creators. They're a completely legitimate and respectable profession.
A lot of people, creators, merchants, etc; often get taken out of business by simple misunderstandings or arbitrary decisions from these platforms. Nobody is going to ruin anything, just have some form of due process where everyone's right to exist is respected.
People imagine Google without ads will be better or paid search makes it better - they are mistaken, since the moment Google (or any top search engines) completely stops serving any ads (or equivalent - everyone using ad block that blocks search ads), all of the companies spending money on ads will instead spend those money on SEO against those search engines. i.e. Selling ads is an effective way to limit the amount of money poured into SEO. As long as the cost of ads is lower than the amount you need to spend on SEO to get the similar amount of traffic through organic results, companies would rather buy ads. This adds more incentive for Google and other search companies to focus on fighting against SEO manipulations also, since a successful SEO effectively lowers the ad price companies are willing to pay. Paid search would make this problem actually worse - people who can afford to pay correlate strongly with the target demographic companies want to show ads, and that means once people starts signing up for paid search, companies will throw more money at SEO.
Now, the doomsday scenario (of SEO winning and all organic results becoming bad) may happen anyway because SEOs end up winning regardless despite Google and other search engines fighting against it. For humanity's sake, we better hope that's not how it ends up. It's less likely given some of the structural advantage search engines have, but SEOs have advantages too, so at this point it's not at all clear one side will necessarily win or lose.
That said I'm not sure I trust the current American political climate to step in and regulate, and I'm not sure they have a right to... The influence of these businesses extends far outside the borders of the United States.
US and EU have different payment process requirements. Same for data collection. There are numerous examples and all platforms find a way to implement them and comply.
Having some creator and merchant protections is not a technical issue. The US could legislate to protect US users and they could comply only in the US. No big deal.
Also note that companies are far from honest in this work - just look at what Google is doing to erode the web platform, and how it is leveraging it's search monopoly to drive businesses towards paying for ads.
I said regulation. You keep your company, but after a certain size, you're obliged to treat the OTHER hard working companies living on your platform with respect and not completely destroy them on a whim.
An example: A friend of mine worked for a battery selling business. They had most of their sales in Amazon. Everything was fine, they had a few years at this and around 100 employees.
One day, Amazon decided one of their papers or whatever wasn't good and closed them down. The appeal process too 3 weeks and bankrupted them.
Amazon was wrong.
Now, you think that's fair? All that hard work and all those people out on the street because some dumbass in amazon's underpaid customer service centers fucked up?
Like I'm saying, there needs to be due process. They shouldn't be able to that.
The problem here is that government doesn't understand technology well enough.
Which makes sense, the majority of people don't. Your post is an example, no disrespect meant.
You can't have transparency in search. It's an arms race between search engines and black hat SEO gaming. It's been that way for decades, likely always will be. That's why the algos absolutely have to be secret, and there has to be no barrier to rapid iteration and innovation. The minute that changes, black hat wins and search goes back to looking like Altavista before it died. In that scenario everyone loses.
It's also imperative that there's a financial incentive to continue to innovate and stay (sort of) ahead in the arms race. It's easy to imagine that a government (or a nonprofit) could run search well, but that's only without an understanding of the scope and depth of the technology involved.
I'm not sure what the best solution is, but there's no question transparency isn't it.
I do, by the way, appreciate your idea of mandating more reasonable due process. However I'd add that if your business relies on search, you're gambling. It's often a smart gamble, for as long as it lasts organic rankings are free money, but be honest with yourself about it.
If Google were to “split up”, I as a customer would like a paid-for, transparant, utility search on one hand. My money can go to dealing with SEO abuse. On the other hand as a business I would like an ad-serving product, to which customers can voluntarily subject themselves to.
The two organizations would have to become separate business units, because I don’t want what’s currently happening: I search for information and get disinformation that others paid for.
Have dataset/s and see where the different technologies, win, fail and IDK.
Solr's Phonetic Matching I always thought needed improvement, it would be interesting to get a report on those for instance.
What metrics would you use? I would be fascinated to know.
Color me interested! There’s so many layers to measure: from natural language understanding to pagerank-style relevance ordering, filtering duplicates to presenting the different permanences of content (breaking news vs books on the same topic).
Yep. It's interesting. How do you deal with versioning. The same document 10 times on a CMS might overpower the search results. Does uniqueness rank higher? There you need a human judge at which search engine / search engine defaults is better.
> natural language understanding
I'd like to know if any search engines do this. I feel like it's out of scope. If they use the words "natural language understanding" I think they are running a scam and mean statistics.
Then prescriptive.
The idea would be a framework so someone can make (or grab their current data) the test different engines for different search terms against the data and mark which is correct or better.
You don't have to 100% agree what search result is better. But to just get to the stage where you can discuss that would be a great start.
And I suspect not agreeing what is better is not as contentious as one thinks.
I also think Google does search real well. So use that for a set point of the truth.