> A new EU project OpenWebSearch.eu … [in which] … the key idea is to separate index construction from the search engines themselves, where the most expensive step to create index shards can be carried out on large clusters while the search engine itself can be operated locally. …[including] an Open-Web-Search Engine Hub, [where anyone can] share their specifications of search engines and pre-computed, regularly updated search indices. … that would enable a new future of human-centric search without privacy concerns.
So.. Who's going to create the index?
Indexing the web is expensive, and its offset by the ads the indexer runs on their search website, such as Google, bing, brave and others.
Every individual website has an incentive to create indices of their own content, and hosting providers could provide it as a service. Not hard to envision. Search Engines could download these indices periodically to build the meta-search.
Lie how? Meta indexes can be picky about the sites they list, simple as that. One could imagine personal meta search engines which only list the sites you care about. Bad sites simply don't get listed.
The point is that if a popular search engine returned pages primarily based on whether particular search terms appeared in their self-published indexes (rather than using a crawler that parsed the page content to build its own index), web sites could easily publish indexes that just listed popular terms users are known to search for regardless of whether the site had anything to do with those search terms. Pretty much what uses to happen 10-15 years ago when everyone believed meta tags were the way to achieve SEO.
Well easy, Europeans are going to fund it with their taxes, public search labs are going to build it, private companies squeeze the concept for cash for as long as possible while taking EU taxpayer money as "research grants" and claiming operational costs as "research tax credits" which will set them up as special parners for the next EU cashgrab (I mean quadriannual plan) since they have successfully run a public-private partnership (into the ground)
t. was involved in one of those plans as part of the research team in a public lab
seems like a very interesting idea. So many times I wanted some kind of advanced gogle-query-language. (i know about allinurl and such, but thats not enough. google is tuned for average user, which is good for google, but not for any non average query)
The real game-changer in search would be if companies would agree to publish indexes of their own sites in an open standard to a place that everyone could access. This would undercut the monopoly power that large search engines have and allow everyone to focus on innovating the best way to search that content vs. having to spend so much time and money to crawl and index it.
I suspect you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for. Even if there were a public, standardised up-to-date high performance full-text index of the entire web freely available I'm willing to bet Google search would be a useful value-add in its ability to answer natural language queries.
> you underestimate how much of the power of search engines is being able to interpret search queries and figure out what a user is really looking for.
So you mean that a search engine is supposed to ignore what you are asking for and instead give you what it thinks you really meant?
If you search for [what year was queen Elizabeth born], Google doesn't return pages that have that phrase in it, rather it returns pages that have an answer to that question. You could call that "ignoring" if you like, but it's what 99% of users expect.
I resort to "natural language queries" only in desperation, when queries that are lists of search-terms have failed.
Actually, they aren't really natural language queries. They are just ordered lists of search-terms. Goo provides no mechanism for saying "This is an english-language question". And even if Goo could parse my natural language, and rephrase it as something like "Are you looking for a list of books published by Douglas Hofstadter?", when you turn that into a query on the index, it stops having anything to do with natural language.
A "dumb" search that just took a search phrase like "books published by Douglas Hofstadter" is going to return pages that have that phrase in it, or something close to that phrase. Google will prioritise results that actually contain such lists, regardless of whether the page contains a phrase like that (e.g. the word "published" is basically ignored by Google).
That's all I meant.
Anyone who thinks this will work has never tried to index a site. A huge amount of effort is spent trying to figure out if the site is serving different content to users vs crawlers, or if the site is coded to appear visually different to humans vs machines. If you ask sites to index themselves you will get lies only.
I index sites all the time and I think it could work. There will be other problems, of course, but we already are partly there with XML sitemaps. Relying on the large search engines to enforce “honesty” from websites puts them into a mediator role that has a number of negative effects both for search in general and, increasingly, society at large.
Relying on sites to be honest about themselves, is even less likely. There are monetary incentives for many of them not to do that. Many sites host dishonest and clickbait content with extreme levels of SEO already. The cost of dishonesty decreases if you can directly modify the index.
I think that is primarily a symptom of the fact that we have a bottleneck on search interface providers. If it were easier / cheaper for new search engines / rankers to exist in the market, they could fairly easily filter out unscrupulous domains.
There are already sitemaps, and pages used structured data like HTML5/ARIA roles, RDF or JSON+LD to provide some semantic annotations.
I'd rather that web robots use this information to build useful indexes than to have to worry about generating yet another feed in the hopes that it helps people find my content in a search engine.
Besides, a web robot can determine how much other sites link to my content and help determine its overall ranking in results. Adding another type of index file to my site will do nothing to determine how it relates to other sites.
The structured data on sites, unfortunately, still requires a crawler to index that content, which serves as a barrier for search engine startups. At a minimum, adding some metadata content to XML sitemaps would go a long way to solving some of this problem (title, meta description, content summary, even structured data to the sitemaps).
What's the problem of using any of the many free webcrawler (libraries) available to crawl a website (even if solely based on the pages advertised by sitemap.xml / robots.txt-announced sitemaps), then extract structured data from these pages?
It's easy to do for small sets of sites, but try doing this at web-scale and you quickly run into a large financial barrier. It's not about technical feasibility as much as it is cost.
Deep down in my soul, the long-locked-away SEO of my money-hustling youth just grinned in anticipation.
We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.
That is indeed the biggest problem but maybe something that can be more effectively dealt with downstream by the content rankers and potentially even the user base / custom search algorithm builders. Brave's Goggles project is a good early prototype of this concept.
Your comment prompted me to check out Searchcode, looks very interesting. How would the federated search model work in this example? Instead of you having to index the various code repositories, they would index themselves and make their search of those indexes available via a federated API?
The market need would then be shifted to the best search interfaces instead of who has the most money to build the biggest index. A much better focus, IMO.
Standard for this already exists [1] but it does not solve the problems of
1. Implementation (sites do not need to have a sitemap; or those that have it, may not have an accurate one)
2. Discoverability (finding sites in the first place, you'll need a centralised directory of all sites; or resort back to crawling in which case sitemaps are not needed)
3. Ranking (biggest problem in creating a search engine)
The sitemaps standard (if this is the basis) would need to be expanded to support additional metadata / structured data to support this idea.
1. This would be up to sites, to your point, major question would be best way to create incentives.
2. This is solvable via a number of approaches, but the search engines themselves would be mostly responsible for finding the right approach for their business. I know how I would do it.
3. Indeed, which would be the main point of this decentralization, to let search engines focus on their hardest problem.
Edit: would Kagi not benefit from having to worry about crawling / indexing sites?
> would Kagi not benefit from having to worry about crawling / indexing sites?
It would, but sitemaps do not provide that function as we discussed above. However if EU Open Web Search succeeded, that is something we could probably use to some extent.
I'm pretty sure we tried that way back in the day with <meta name="keywords" content="spam spam spam spam">. People would stuff that with every word in the english language. Older search engines that used those keywords returned some pretty awful results. You simply can't trust sites, who have a strong incentive to get to the top of SEO rankings, to not lie. In fact, given at least one of your competitors will stuff their keywords to get to the top you'll have to do it too. It would become an arms race for who can stuff the most garbage into their indexes to "win". It just doesn't work.
All search engines that attempt to be useful will have to filter out the junk. You just have to trust that the search engine you are using isn't withholding results from you that it considers "bad" (eg: "misinformation" (i.e. stuff somebody disagrees with)).
And to me, that is the crux of the debate really. Nobody wants spam for search results--everybody agrees with that and there is no real debate about filtering that crap out. The argument really is should a very large company that has a huge market share get to decide what constitutes "fact" and what is "misinformation". Based on 2.5 years of experience so far, what was once deemed "misinformation" has a sneaky way of becoming "factual information". Labeling and hiding "misinformation" because it goes against some narrative pushed by incredibly powerful entities is very scary and there was a hell of a lot of exactly that going on during this covid crap.
I used to fall on the side of "private companies can do whatever they want" but now I'm not so sure. Companies like FB, Twitter or Google play a huge role in shaping politics and society. I'm no longer convinced it is okay to let them play the role of "fact checker" or anything like that. Filtering spam is one thing, but hiding "misinformation" is entirely different.
Your last point is also the one (aside from the economics) I am the most interested in.
I think we live in a world now where we are so used to a few tech giants mediating everything for us that we can't even imagine other solutions to this problem, but it's also how we got to this point in the first place.
You need a trustworthy core by which you can judge the vote of new users. You can incorporate them until somebody complains about a result that is out of place.
This doesn't have to fully scale. There are many pages without monetary value that won't be manipulated. The tags are an additional signal that can be used where they work. If they don't work, they can be ignored.
But it will scale because there are far more consumers than producers.
I was a heavy AltaVista user back in the time, I can tell you Google results were really higher quality, it wasn't only their homepage (even if that was also quite refreshing).
I suspect search engines are an outdated concept for at least the largest of sites, who will generally, but not always, have better ways to directly search their own content.
The remainder of the search problem seems to just be collecting relevant trafficked sites for listing in results. Today Google et al seem to be doing this BY HAND. And it's not even obfuscated.
Recently, for the first time in my life, the wizard behind the curtain seems to have been exposed. I feel strongly that one could probably start a small index that catered to a fairly large audience.
And honestly, for other queries, just tell the user to search that site directly. I think you could even market it to users as not a technical limitation, but behavior that should be considered fuddy-duddy.
Like, really, you're going to search me? You know they have their own search right?
Even Yellow Pages faded into obscurity eventually.
I would agree; I've almost never seen a site that provides a better way to directly search its own content than the major search engines provide. Wikipedia might come closest because their search compares the user input to article titles, and that's often helpful (mainly because the article titles are chosen to describe the article content, and aren't clickbait). But pretty much anything else uses a search approach that is far inferior to what Google or Bing provide, and it won't find what you want.
That cannot be true, as the project has yet to start. But anyone can start a crawler, so you may have encountered other people's software. We wouldn't be so unknowledgeable to ignore robots.txt ;-)
It was a crawler with the user agent "hgf AlphaXCrawl/0.1 (+https://www.fim.uni-passau.de/data-science/forschung/open-se...)", operated by the University Passau and Open Search Foundation named on your landing page. It would be a mighty big coincidence if this wasn't a project connected to this endeavor, especially when it confirms being an experimental crawler of said project at the UA URL.
Storage or not is entirely irrelevant to robots.txt directives. It guides automated access. It must be parsed first and excluded URLs must not be accessed at all.
It's definitely a comparative underdog regardless, but if you think Alphabet spends anywhere near 40% on search you're out of your mind. I'd be shocked if their spend is double-digits. I'd be unsurprised if it's <1%.
the EU loves taxing productive companies and wasting said money in stillborn projects that nevertheless promise a kind of bright socialist federalist Europe in their bureaucratic minds
At least we have some of the most livable countries on Earth to show for it. I take taxes over any trickle-down economics, and don't let me stop you looking up the definition of socialist, because you are using it wrong.
Besides it's a 8.5 million EUR project, it's literally nothing, it's payroll for a few people. The money is being invested into people who then spend most of it, so it's a triple investment.
Slovenia, Czech Republic. But yes, I think there was a competing proposal from Italy/Spain. Not enough budget for two projects in this area, unfortunately, as they were good too.
I have written this before but I’ll put it here again. What I would like to see is a federated search engine. Based on activitypub that works like mastodon. Don’t like the results from one source? Just remove them from your sources, or lower their ranking. Similar to yacy but you can work with the protocol to connect or build whatever type of index you want using whatever technology you like, and communicate over an existing standard. Want to build the worlds best index of Pokémon sites, then go do it. Want to build a search engine using idris or ats? Sure! I did note the professors are on mastodon so perhaps this may actually happen.
One of these days I’ll actually implement the above assuming nobody else does. I figured if I can at least get the basics done and a reference implementation that’s easy to run it could prove the concept. If anyone is interested in this do email my in my bio.
What I worry about for this project is that it becomes another island which prohibits remixing of results like google and bing, and its own index and ranking algorithms become gamed.
I wish the creators best of luck though. I am also hoping for some more blogs and papers about the internals of he engine. So little information is published in the space that anything is welcome, especially if it’s deeply technical.
We like Federated search, we like decentralized search, and even P2P search; we are trying to find a good mix, and decided to get started rather than wait! Exciting times.
I would love to be able to run a node that mirrors part or all of an index like this, and to let people query it - a bit like https://torrents-csv.ml/#/
Good luck! I'll be watching your progress and cheering you all on!
I'm not trying to be dismissive, it's just my feeling from working on search.marginalia.nu is that nearly every aspect of search benefits from locality, not only is the full crawl-set instrumental in determining both domain rankings and relevance signals on a term-level such as anchor tag keywords; but the way an inverted index is typically set up is extremely disk cache friendly where the access pattern for checking the first document warms up the cache for the other queries, but that discount obviously only exists when it's the same cache.
You could get people creating indexes with love such as your own. marginalia could become the de-facto index for long form content. However you probably arent that interested in running the best pokemon website, so someone else could do that.
Enough people add domain specific search endpoints, with perhaps a taxonomy to say "hey send those sort of queries over here" and you have a compelling engine that self heals should someone stop running things, or starts spamming.
You can also integrate search results for which you cannot have the index, like social media APIs, another reason.
You could also mix and match search results from various topic-oriented indices. That's a research question, whether that is really better than building one unified one. But we think it is the way to bring index fragments to the edge, with the obvious privacy advantages.
What benefit does federation bring here? Unless it is very simple to set up, most communities are non-technical and probably won't be able to set up their own crawler. I would think just a search engine that lets you customize the ranking algorithm, and maybe hook into whatever ontology they've developed and ranking it accordingly would be sufficient.
It can be very simple to setup. Think single binary to run, or lambda to deploy (yes this is possible) with the URL back to it.
I imagine a binary, with a simple Admin UI allowing you to crawl some domains recursively would be enough to index your own website, and then have those results shared.
Where I could see this being really useful, is let someone who knows everything about pokemon provide the index for searching pokemon information. Then when they federate, provide a taxonomy saying "for queries that have these words, call me". Suddenly you have a very high value search source for pokemon.
Throw in some zero click info information boxes and you have added a lot of value.
> most communities are non-technical and probably won't be able to set up their own crawler
They can use a solution which already integrates the search. Forums and CMSes are a good target for that. Then you can say "I'd like my search to look at widgetlovers.com too" - and you get their sitemap + featured external links, because they run FooPress that supports it.
Kind of the same as sitemap we already produce for Google.
> Don’t like the results from one source? Just remove them from your sources, or lower their ranking.
That's basically Usenet killfiles and, yes, I think they're totally due for a comeback in one form or another. Usenet may have had its issues towards the end (although it still exists), but killfiles weren't one of its problems. The simplest one you could just discard sources you didn't want to read anymore but the more advanced you could assign weight/rankings based on various factors (keywords / usernames / if you did participate or not in a discussion / etc.).
One of the things I wonder here is if it would be easier to just start by crawling known RSS feeds and then exposing a JSON API for the data and making the whole thing open source. Then keeping a public list of indexes and who crawls what. Eventually moving into crawling other sources but first primarily addressing the majority of useful content that's easily parseable.
Isn't searx what you're describing? I was running an instance for a while, and it's basically a meta search engine that has support for all kinds of providers.
There are also some web extensions available so that you can fill it with more data.
If it is about a decentralized index there's also YaCy.net [1] but I don't know how actively maintained the project is.
For me it made more of an enterprise-grade use case (e.g. for building a search for your own file servers or confluence) so I only tested it out a little. It's a huge java project, that's why I decided to go with searx back then...cause yacy was pretty hard to setup.
What's the point of a federated search engine? At the end of the day most nodes will end up implementing the same regulations/censorship with development driven primarily by a few. It's like ethereum vs ethereum classic all over again. If the EU or the developers' respective governments demand a censorship or forgetting feature to be implemented, it's not like the federated nature would matter. An open source search index is useful, a search engine that can be easily self hosted is also useful. But building a search engine as a federated system is a gimmick with no significant value.
Do you see any major Mastodon nodes interfacing with Truth Social or Gab? I certainly don't. If federation barely works for a social media app, I fail to see how it would even matter for a search engine.
ActivityPub is not well suited for this application. It's for publishing activities made by actors — hence the name. You'll want to invent your own federation protocol specifically for federated search.
I'd really like to see them match the 20+ years of search quality fine-tuning that Google built into their search engine.
Not that Google is as good as it used to, but still, catching up with them is way more complicated than just building a big crawl + index piece of infrastructure.
And all of that on a government-funded shoestring budget.
I'd like to see them match the 10 years of fine tuning from 2012 google.
My understanding is google used to keep a lot of data around for "long tail" queries, and stopped doing that at some point. This seems to be the issue around search declining outside of "food near me" type queries (which they absolutely excel at).
so it began, that sern starts to gather market share.
--
I doubt this will take off. I mean they investend more in funding and marketing instead of starting to built something.
they should've started with code (agpl3 of course) and invited more and more people. at the moment this is more buzzword bingo bullshit than anything else. it's basically always the same problem, instead of focusing on the product, they fous more on the message.
There's no such thing as an unbiased yet useful search engine. The very act of choosing which article to put at the top is a value judgment, therefore a bias. The only way to make things "unbiased" would be to rank the matches randomly, and that would be useless.
So would the SEO people. But even disregarding that problem, to have a halfway decent search ranking you're going to need deep learning, and you'll get a model that is largely opaque.
When I said I wanted control over the ranking, I meant as a user. Tell me what parameters I can rank on; give me a UI that makes it reasonably easy to express a ranking. That's it.
I hope UI design and stuff is orthogonal to the construction of an open index. I think this could be very interesting.
What does "based on European values and jurisdiction" refer to? I'd love to be pleasantly surprise, but this sounds like it's ripe for centralized censorship.
Given the history of the 20th century, this kind of comment promoting European values and jurisdiction seems..... dicey. Companies ethical records, as shitty as they are have nothing on the mass destruction, genocide and stupidity of governments.
Um, bad example. It was a misconceived bit of legislation. There's no logic behind that claimed right. What happened to my right to remember what I choose? Why shouldn't I publish my memories on a website?
Anyway, it's not a "European value", it's a piece of legislation, so it falls under the "European jurisdiction" category, not the "European values" category.
That complying with "European values" is part of the project goals is unfortunate. Who decides what are European values? There's no Declaration of European Values. We don't all have the same values. There are just European laws, and then a whole bunch of opinions that not all Europeans agree about.
people really, truly, go out of their way to criticise the EU. there are things about the EU that deserve criticism, but this is reaching to the point of deliberate misunderstanding. no one is saying that a person who wants to be forgotten cannot be talked about on your website, or that you have to somehow deliberately delete your memories of that person. you are not a corporation. the right to be forgotten is about the right for you to be forgotten by a corporation. the right for you to not be in their database anymore
It could be fun to have a conversation about this topic, but when you start with calling the opinions of others a "deliberate misunderstanding", you've poisoned the well.
Still, it's a good illustration that it's a somewhat unique European value. It's definitely not an American value.
> the right for you to not be in their database anymore
Let's switch that around: you're referring to their right not to be listed in my database any more.
I don't see what difference it makes whether I'm incorporated or not; I'm not OK with the idea that certain information cannot be shared. Like, if the information is false, that's one thing; but a legal requirement that some true information must be hidden, that seems like pure badness.
your argument is predicated on the assumption that there’s no difference between you and a large group of people. this is very clearly a false assumption
By US values profit is weighted heavily over anything else. Chinese and Russian values (or lack thereof) are focused on controlling the narrative. European values would contrast as rules (as opposed to vague guidelines) with legal ramification, a lot of open source software, an actual effort towards honesty (mileage may vary) and probably vogon poetry.
> Chinese and Russian values (or lack thereof) are focused on controlling the narrative
I think it's reasonable to label China and Russia as control freaks, but the US and Europe aren't much different at this point too given the creepy focus on trying to shut down whatever they want by calling it "misinformation" and pressuring social media to comply.
I'm a bit skeptical EU-funding a bunch of professors is the way a search engine will be built.
The primary goal for academics is to publish new findings, while what you need to build a search engine is rock solid CS and information retrieval basics. Academically, it's not very exciting. Most of it was hashed out in the 1980s or earlier.
I don't think the number of people or even the size of the budget is wrong. A small team can be incredibly powerful and productive if you have the right people. In fact, I think far more often search engines fail from trying to start too big than too small.
The problem is that you need people who actually know how to architect complex software systems much more than you need revolutionary new algorithms. For that, professors are the wrong people. A professor on the team, sure, that might be helpful. Not half a Manhattan project's worth.
It happens all the time in Europe. Collaboration between public and private companies is pretty much a pipe dream in the EU. Some company that actually works on building search technology would achieve way more than a bunch of professors.
I disagree on the budget though. It is basically pocket change.
Arguably the biggest most unsolved problem in search is how to make a profit (or even break even). This can be approached in two ways: You can either try to find some way of making search more profitable, or you can find a way to make search cheaper. I think the latter is a lot more plausible than the former.
A shoestring budget keeps the costs down by design and by necessity. A large budget virtually ensures the search engine becomes so expensive to operate it will never break even.
Why not offer a paid tier? Seems to work for Kagi. Information elites will soon flock to paid search engines, which won’t be much more expensive than a streaming subscription. I pay for Netflix and am willing to pay for a search engine that offers as good a search service as the video streaming offered by Netflix.
Have no fear; all of the actual work will be done by PhD students straight out of undergrad, and most of the actual leadership will be done by a string of recent PhD grads who need results in 6 months because they'll be full time job marketing for the 6 months after that ;-)
Who deliberately did not stay in academia to do it.
More to the point, a successful team building a product like a search engine requires roles that academia doesn't really have.
Who is doing product management?
Who is doing product marketing?
etc
This is all applied engineering at this point, not R&D. How does it at all fit into academia's strong suit?
I think that maybe the point is this is not being tackled as a purely economic endeavour (or if it is, it’s in the “indirect” manner), as such, I suspect roles like “product marketing” are probably unnecessary, at least for now.
Also, tell me you wouldn’t love to work on a large project that wouldn’t be subject to the arbitrary whims and promises of the marketing department.
> the point is this is not being tackled as a purely economic endeavour (or if it is, it’s in the “indirect” manner), as such, I suspect roles like “product marketing” are probably unnecessary, at least for now
Which results in an interesting engine nobody uses. Products that start with the tech and then think of selling it fall on their faces for a reason.
They weren't forced to pivot. Gates explained to they how much money is to be made and they changed their mind. For some time they were very excited how unobtrusive and helpful the ads were. Then they realised there's even more money and the rest is history.
Is Google proves anything, it's that greed is real.
>I'm a bit skeptical EU-funding a bunch of professors is the way a search engine will be built.
Heh, so, funny story...
>A second grant—the DARPA-NSF grant most closely associated with Google’s origin—was part of a coordinated effort to build a massive digital library using the internet as its backbone. Both grants funded research by two graduate students who were making rapid advances in web-page ranking, as well as tracking (and making sense of) user queries: future Google cofounders Sergey Brin and Larry Page.
>The research by Brin and Page under these grants became the heart of Google: people using search functions to find precisely what they wanted inside a very large data set.
I see this a lot in a project I'm involved with where the majority of contributors are from academia.
The incentive is to mostly push novel things, often with questionable practicability, in order to write a paper about it.
But nobody will implement the 'boring' features needed to make the thing generally useful.
The best remote+paying job that I've ever seen online was from a Darpa project (memex project, about search engines). $180K - $250K+. This was ~5 years ago.
The project can fail, you are correct, but it does not take anything from any other projects, it is just an other initiative trying to contribute in the space.
Parent commenter own search project Marginalia Search [1] could even benefit from it, or even maybe collaborate with it.
It is not a winner-take-all situation, and we need various open initiative in this space to get out of the current conundrum we are in with Google stronghold on search.
Taxes are taking from funds for personal projects, and universities already get a lot of funding in Europe to do research, maybe they should focus on creating a better environment for students to become researchers or entrepreneurs instead.
Overall I think there are better ways to improve search from an EU perspective by doing what they are supposed to be doing:
- create a fair environment for companies to compete in, e.g., take a look how Google, Apple or Meta's assets are set up to make it harder for competitors, break that up
- improve standards in eduction – it doesn't really make sense for all member countries to think of and maintain a good CS curriculum and they all seem to be pretty bad at it
- make it easier to build something and get funded, and reward creating prosperity, don't tax it to death
Just tested marginalia's random mode btw. Pretty cool, reminds me of the internet when I was a kid
CIA? Other people are saying DARPA and NASA got together to fund the NSF which funded the PhDs of the founders of Google, but even that's a bit too indirect IMO. Where does the CIA fit in?
(Plus: for who might not know, DARPA is US defense research, and heavily influenced by the intelligence services needs. Which is not necessarily bad! Just good to understand where and how Google originated. And wrt DARPA, they funded the creation of the internet itself, for whatever matters.
In Europe, things often go slightly different. The Web is a result of CERN, who are also a project partner of OpenWebSearch.EU. Why? Well, better search can also be beneficial for better science, not just for end users wanting to find their way or buying something.)
I salute your efforts and endorse your search engine. I also recognize that you know what it takes to build a search engine.
I don't think you have deep familiarity with EU academia.
- The primary goal for academia is to influence society. Publishing is a route to that.
- Being head of the EU search engine project would give high academic status
- There are hundreds of articles which you could publish on this project
- Rock solid CS. Would someone like Knuth count as "rock-solid"? Who is better at CS, the person who can implement quicksort because they practiced leetcode, or the person who invented quicksort?
- information retrieval basics. Again, these basics were probably developed in academia.
The skills you say are basic to this endevour are more prevalent in top-quality professors and post-docs than they are in industry.
On top of these skills, you actually need most of all software engineering and architecture experience. I don't think this is common in academia at all. Not in professors, not in PhD-students. You need practical experience building complex software at a large scale. Across that, you need to implement these CS fundamentals.
This is requires far more CS than you'll find in your usual software development effort, for sure, and many CS professors absolutely fit that bill. However, to the same degree it also demands far more on the software engineering side. People out of academia in general, from every time I've seen them build software, have not been all too impressive on that side of things.
Web search has an incredible demand for being well rounded, beyond anything else I've encountered. CS isn't the hard part bottle-necking everything else, it's just one of the many hard parts.
On first glance, I see the word "unbiased" immediately followed by "based on European values". Now, I'm no expert, but to me, that seems pretty biased.
"being unbiased" can be fairly honestly called a "value".
Europeans apparently have chosen it as one of their own values.
If you are arguing that being truly unbiased cannot ever be realized, I counter that neither truth nor justice can ever be truly realized, by this should not stop anyone from taking them as their values.
It's not that there's anything wrong with aspiring to be unbiased; the problem arises when someone really thinks they are unbiased, or claims to be unbiased. You get this a lot in the MSM; the Guardian constantly exhorts it's readers to subscribe, to support unbiased journalism.
I prefer reporting that wears its biases on its sleeve. Regrettably The Guardian squeezed out most of its most interesting writers, apparently by repeatedly spiking their stories.
Yeah, Truth and Justice are platonic abstracts; nobody thinks they exist in the real world. But people do believe that unbiased reporting is possible. It isn't.
326 comments
[ 4.7 ms ] story [ 339 ms ] threadSo.. Who's going to create the index? Indexing the web is expensive, and its offset by the ads the indexer runs on their search website, such as Google, bing, brave and others.
Bullseye.
Regarding privacy the bar is significantly higher than what Google has to deal with. This will come at some cost in quality and/or speed.
t. was involved in one of those plans as part of the research team in a public lab
The actual website of the project (with some concrete info) can be found here: https://openwebsearch.eu/
So you mean that a search engine is supposed to ignore what you are asking for and instead give you what it thinks you really meant?
Actually, they aren't really natural language queries. They are just ordered lists of search-terms. Goo provides no mechanism for saying "This is an english-language question". And even if Goo could parse my natural language, and rephrase it as something like "Are you looking for a list of books published by Douglas Hofstadter?", when you turn that into a query on the index, it stops having anything to do with natural language.
Not only are some sites malicious -- mostly unimportant ones -- but many good sites are simply incompetent.
I'd rather that web robots use this information to build useful indexes than to have to worry about generating yet another feed in the hopes that it helps people find my content in a search engine.
Besides, a web robot can determine how much other sites link to my content and help determine its overall ranking in results. Adding another type of index file to my site will do nothing to determine how it relates to other sites.
I don't see this as a barrier unique to startups.
We have had embedded metadata in websites for decades. In the beginning, Search Engines did even use them. Until someone started stuffing unrelated keywords in it to rank higher.
How is that any different from requiring a crawler to index XML sitemaps?
> At a minimum, adding some metadata content to XML sitemaps
The purpose of a sitemap is to tell a web robot what resources there are, with some minimal metadata about page titles and last modified date.
Google has some extensions for identifying images and videos.
But that adds more work for site maintainers, who have to duplicate work.
1. Implementation (sites do not need to have a sitemap; or those that have it, may not have an accurate one)
2. Discoverability (finding sites in the first place, you'll need a centralised directory of all sites; or resort back to crawling in which case sitemaps are not needed)
3. Ranking (biggest problem in creating a search engine)
[1] https://www.sitemaps.org/protocol.html
1. This would be up to sites, to your point, major question would be best way to create incentives.
2. This is solvable via a number of approaches, but the search engines themselves would be mostly responsible for finding the right approach for their business. I know how I would do it.
3. Indeed, which would be the main point of this decentralization, to let search engines focus on their hardest problem.
Edit: would Kagi not benefit from having to worry about crawling / indexing sites?
It would, but sitemaps do not provide that function as we discussed above. However if EU Open Web Search succeeded, that is something we could probably use to some extent.
All search engines that attempt to be useful will have to filter out the junk. You just have to trust that the search engine you are using isn't withholding results from you that it considers "bad" (eg: "misinformation" (i.e. stuff somebody disagrees with)).
And to me, that is the crux of the debate really. Nobody wants spam for search results--everybody agrees with that and there is no real debate about filtering that crap out. The argument really is should a very large company that has a huge market share get to decide what constitutes "fact" and what is "misinformation". Based on 2.5 years of experience so far, what was once deemed "misinformation" has a sneaky way of becoming "factual information". Labeling and hiding "misinformation" because it goes against some narrative pushed by incredibly powerful entities is very scary and there was a hell of a lot of exactly that going on during this covid crap.
I used to fall on the side of "private companies can do whatever they want" but now I'm not so sure. Companies like FB, Twitter or Google play a huge role in shaping politics and society. I'm no longer convinced it is okay to let them play the role of "fact checker" or anything like that. Filtering spam is one thing, but hiding "misinformation" is entirely different.
I think we live in a world now where we are so used to a few tech giants mediating everything for us that we can't even imagine other solutions to this problem, but it's also how we got to this point in the first place.
Why is it not enough to punish sites that abuse the keywords?
You need a trustworthy core by which you can judge the vote of new users. You can incorporate them until somebody complains about a result that is out of place.
This doesn't have to fully scale. There are many pages without monetary value that won't be manipulated. The tags are an additional signal that can be used where they work. If they don't work, they can be ignored.
But it will scale because there are far more consumers than producers.
What made Google such a game changer was that they based their index not just on the contents, but on how pages linked to each other.
The remainder of the search problem seems to just be collecting relevant trafficked sites for listing in results. Today Google et al seem to be doing this BY HAND. And it's not even obfuscated.
Recently, for the first time in my life, the wizard behind the curtain seems to have been exposed. I feel strongly that one could probably start a small index that catered to a fairly large audience.
And honestly, for other queries, just tell the user to search that site directly. I think you could even market it to users as not a technical limitation, but behavior that should be considered fuddy-duddy.
Like, really, you're going to search me? You know they have their own search right?
Even Yellow Pages faded into obscurity eventually.
I have the exact opposite experience.
To wit: searching HN via the algolia link at the bottom is way worse than searching on Google with a site:ycombinator.com restrict.
Same thing for YouTube, where the search engine is tuned for maximizing watch time and strictly not to return what you're looking for.
Just because it crawled it doesn't mean it stored it.
> Resource Limit Is Reached
> The website is temporarily unable to service your request as it exceeded resource limit. Please try again later.
Original URL might be more resilient...
https://web.archive.org/web/20220920183027/https://openwebse...
How much of what google spends on "search" is strictly for search, vs business goals related to search?
How much of that google spend is salary? What are those salaries? How do they compare with EU post-doc salaries?
Besides it's a 8.5 million EUR project, it's literally nothing, it's payroll for a few people. The money is being invested into people who then spend most of it, so it's a triple investment.
No research institutes from {France, Italy, Spain, Greece, Portugal, etc ...} involved.
[0] https://en.wikipedia.org/wiki/Quaero
[1] https://www.dw.com/en/germany-pulls-away-from-quaero-search-...
We are very aware of the Quaero/Theseus history :-)
These search engines will then have the freedom to define their own search product experience, business model, even ranking of results.
One of these days I’ll actually implement the above assuming nobody else does. I figured if I can at least get the basics done and a reference implementation that’s easy to run it could prove the concept. If anyone is interested in this do email my in my bio.
What I worry about for this project is that it becomes another island which prohibits remixing of results like google and bing, and its own index and ranking algorithms become gamed.
I wish the creators best of luck though. I am also hoping for some more blogs and papers about the internals of he engine. So little information is published in the space that anything is welcome, especially if it’s deeply technical.
Good luck! I'll be watching your progress and cheering you all on!
I'm not trying to be dismissive, it's just my feeling from working on search.marginalia.nu is that nearly every aspect of search benefits from locality, not only is the full crawl-set instrumental in determining both domain rankings and relevance signals on a term-level such as anchor tag keywords; but the way an inverted index is typically set up is extremely disk cache friendly where the access pattern for checking the first document warms up the cache for the other queries, but that discount obviously only exists when it's the same cache.
Enough people add domain specific search endpoints, with perhaps a taxonomy to say "hey send those sort of queries over here" and you have a compelling engine that self heals should someone stop running things, or starts spamming.
You can also integrate search results for which you cannot have the index, like social media APIs, another reason.
You could also mix and match search results from various topic-oriented indices. That's a research question, whether that is really better than building one unified one. But we think it is the way to bring index fragments to the edge, with the obvious privacy advantages.
I imagine a binary, with a simple Admin UI allowing you to crawl some domains recursively would be enough to index your own website, and then have those results shared.
Where I could see this being really useful, is let someone who knows everything about pokemon provide the index for searching pokemon information. Then when they federate, provide a taxonomy saying "for queries that have these words, call me". Suddenly you have a very high value search source for pokemon.
Throw in some zero click info information boxes and you have added a lot of value.
They can use a solution which already integrates the search. Forums and CMSes are a good target for that. Then you can say "I'd like my search to look at widgetlovers.com too" - and you get their sitemap + featured external links, because they run FooPress that supports it.
Kind of the same as sitemap we already produce for Google.
That's basically Usenet killfiles and, yes, I think they're totally due for a comeback in one form or another. Usenet may have had its issues towards the end (although it still exists), but killfiles weren't one of its problems. The simplest one you could just discard sources you didn't want to read anymore but the more advanced you could assign weight/rankings based on various factors (keywords / usernames / if you did participate or not in a discussion / etc.).
There are also some web extensions available so that you can fill it with more data.
[1] https://searx.github.io/searx/
For me it made more of an enterprise-grade use case (e.g. for building a search for your own file servers or confluence) so I only tested it out a little. It's a huge java project, that's why I decided to go with searx back then...cause yacy was pretty hard to setup.
[1] https://github.com/yacy
> a decentralized p2p websearch and collaborative tool.
> It relies on a distributed collaborative filter[6] to let users personalize and share their preferred results on a search.
Do you see any major Mastodon nodes interfacing with Truth Social or Gab? I certainly don't. If federation barely works for a social media app, I fail to see how it would even matter for a search engine.
Even so you could base it on activitypub I suspect. It would need to be extended for sure to implement the sorts of things believe would be required.
I'd really like to see them match the 20+ years of search quality fine-tuning that Google built into their search engine.
Not that Google is as good as it used to, but still, catching up with them is way more complicated than just building a big crawl + index piece of infrastructure.
And all of that on a government-funded shoestring budget.
Mmmh.
Good luck to them, but I'm not holding my breath.
My understanding is google used to keep a lot of data around for "long tail" queries, and stopped doing that at some point. This seems to be the issue around search declining outside of "food near me" type queries (which they absolutely excel at).
--
I doubt this will take off. I mean they investend more in funding and marketing instead of starting to built something. they should've started with code (agpl3 of course) and invited more and more people. at the moment this is more buzzword bingo bullshit than anything else. it's basically always the same problem, instead of focusing on the product, they fous more on the message.
Well, I would rather like to have visibility into the ranking algorithm, and to be able to control the resultset ordering.
When I said I wanted control over the ranking, I meant as a user. Tell me what parameters I can rank on; give me a UI that makes it reasonably easy to express a ranking. That's it.
I hope UI design and stuff is orthogonal to the construction of an open index. I think this could be very interesting.
Anyway, it's not a "European value", it's a piece of legislation, so it falls under the "European jurisdiction" category, not the "European values" category.
That complying with "European values" is part of the project goals is unfortunate. Who decides what are European values? There's no Declaration of European Values. We don't all have the same values. There are just European laws, and then a whole bunch of opinions that not all Europeans agree about.
Still, it's a good illustration that it's a somewhat unique European value. It's definitely not an American value.
do you also think you’re not allowed to remember people’s names without permission because of GDPR?
Let's switch that around: you're referring to their right not to be listed in my database any more.
I don't see what difference it makes whether I'm incorporated or not; I'm not OK with the idea that certain information cannot be shared. Like, if the information is false, that's one thing; but a legal requirement that some true information must be hidden, that seems like pure badness.
I think it's reasonable to label China and Russia as control freaks, but the US and Europe aren't much different at this point too given the creepy focus on trying to shut down whatever they want by calling it "misinformation" and pressuring social media to comply.
The primary goal for academics is to publish new findings, while what you need to build a search engine is rock solid CS and information retrieval basics. Academically, it's not very exciting. Most of it was hashed out in the 1980s or earlier.
> 7 countries.
> 25+ people.
There are literally dozens of them!
https://openwebsearch.eu/partners/
The problem is that you need people who actually know how to architect complex software systems much more than you need revolutionary new algorithms. For that, professors are the wrong people. A professor on the team, sure, that might be helpful. Not half a Manhattan project's worth.
I disagree on the budget though. It is basically pocket change.
A shoestring budget keeps the costs down by design and by necessity. A large budget virtually ensures the search engine becomes so expensive to operate it will never break even.
And the EU just solved that problem.
Have no fear; all of the actual work will be done by PhD students straight out of undergrad, and most of the actual leadership will be done by a string of recent PhD grads who need results in 6 months because they'll be full time job marketing for the 6 months after that ;-)
Who is doing product management?
Who is doing product marketing?
etc
This is all applied engineering at this point, not R&D. How does it at all fit into academia's strong suit?
Also, tell me you wouldn’t love to work on a large project that wouldn’t be subject to the arbitrary whims and promises of the marketing department.
Which results in an interesting engine nobody uses. Products that start with the tech and then think of selling it fall on their faces for a reason.
Is Google proves anything, it's that greed is real.
Heh, so, funny story...
>A second grant—the DARPA-NSF grant most closely associated with Google’s origin—was part of a coordinated effort to build a massive digital library using the internet as its backbone. Both grants funded research by two graduate students who were making rapid advances in web-page ranking, as well as tracking (and making sense of) user queries: future Google cofounders Sergey Brin and Larry Page.
>The research by Brin and Page under these grants became the heart of Google: people using search functions to find precisely what they wanted inside a very large data set.
https://qz.com/1145669/googles-true-origin-partly-lies-in-ci...
But nobody will implement the 'boring' features needed to make the thing generally useful.
Curious what the salaries will be on this one.
I’m just skeptical if the EU bureaucrats will put the money in the right place, and if this is even the right approach.
Parent commenter own search project Marginalia Search [1] could even benefit from it, or even maybe collaborate with it.
It is not a winner-take-all situation, and we need various open initiative in this space to get out of the current conundrum we are in with Google stronghold on search.
1: https://search.marginalia.nu
Overall I think there are better ways to improve search from an EU perspective by doing what they are supposed to be doing:
- create a fair environment for companies to compete in, e.g., take a look how Google, Apple or Meta's assets are set up to make it harder for competitors, break that up
- improve standards in eduction – it doesn't really make sense for all member countries to think of and maintain a good CS curriculum and they all seem to be pretty bad at it
- make it easier to build something and get funded, and reward creating prosperity, don't tax it to death
Just tested marginalia's random mode btw. Pretty cool, reminds me of the internet when I was a kid
(edit: formatting)
Enjoy! It's a great story.
(Plus: for who might not know, DARPA is US defense research, and heavily influenced by the intelligence services needs. Which is not necessarily bad! Just good to understand where and how Google originated. And wrt DARPA, they funded the creation of the internet itself, for whatever matters.
In Europe, things often go slightly different. The Web is a result of CERN, who are also a project partner of OpenWebSearch.EU. Why? Well, better search can also be beneficial for better science, not just for end users wanting to find their way or buying something.)
I salute your efforts and endorse your search engine. I also recognize that you know what it takes to build a search engine.
I don't think you have deep familiarity with EU academia. - The primary goal for academia is to influence society. Publishing is a route to that. - Being head of the EU search engine project would give high academic status - There are hundreds of articles which you could publish on this project - Rock solid CS. Would someone like Knuth count as "rock-solid"? Who is better at CS, the person who can implement quicksort because they practiced leetcode, or the person who invented quicksort? - information retrieval basics. Again, these basics were probably developed in academia.
The skills you say are basic to this endevour are more prevalent in top-quality professors and post-docs than they are in industry.
This is requires far more CS than you'll find in your usual software development effort, for sure, and many CS professors absolutely fit that bill. However, to the same degree it also demands far more on the software engineering side. People out of academia in general, from every time I've seen them build software, have not been all too impressive on that side of things.
Web search has an incredible demand for being well rounded, beyond anything else I've encountered. CS isn't the hard part bottle-necking everything else, it's just one of the many hard parts.
Worked for Google.
If you are arguing that being truly unbiased cannot ever be realized, I counter that neither truth nor justice can ever be truly realized, by this should not stop anyone from taking them as their values.
I prefer reporting that wears its biases on its sleeve. Regrettably The Guardian squeezed out most of its most interesting writers, apparently by repeatedly spiking their stories.
Yeah, Truth and Justice are platonic abstracts; nobody thinks they exist in the real world. But people do believe that unbiased reporting is possible. It isn't.