Great job with Marginalia. Do you plan to open source your data as well or only the code?
Thanks! The active index is running on 4 servers and we have one server for hosting the frontend and the api (the API is what is used by the frontend, ex: https://api.alexandria.org/?q=hacker%20news) Then we have one…
This is actually not used yet. Working on implementing that as a factor.
We are currently just doing an intersection and then we make a lookup in a forward index to get the urls, titles and snippets. I actually don't know what roaring bitmaps are, please enlighten me :)
Founder here. We are using harmonic centrality instead of pagerank. But of course much more work needs to be done to make the search engine usable.
Yes it would be impossible to keep the index in RAM. >> It's hard for me to see how that could be done much faster unless you find a way to parallelize the process We actually parallelize the process. We do it by…
Hi, Yes our documentation is probably pretty confusing. It works like this, the base score for all URLs to a specific domain is the harmonic centrality (hc). Then we have two indexes, one with URLs and one with links…
Founder here, I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost all big websites are behind cloudflare…
The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost. It…
Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are…
Interesting, in Sweden I only got this story when I made the same searches: https://imgur.com/a/k1Avbtm
Great job with Marginalia. Do you plan to open source your data as well or only the code?
Thanks! The active index is running on 4 servers and we have one server for hosting the frontend and the api (the API is what is used by the frontend, ex: https://api.alexandria.org/?q=hacker%20news) Then we have one…
This is actually not used yet. Working on implementing that as a factor.
We are currently just doing an intersection and then we make a lookup in a forward index to get the urls, titles and snippets. I actually don't know what roaring bitmaps are, please enlighten me :)
Founder here. We are using harmonic centrality instead of pagerank. But of course much more work needs to be done to make the search engine usable.
Yes it would be impossible to keep the index in RAM. >> It's hard for me to see how that could be done much faster unless you find a way to parallelize the process We actually parallelize the process. We do it by…
Hi, Yes our documentation is probably pretty confusing. It works like this, the base score for all URLs to a specific domain is the harmonic centrality (hc). Then we have two indexes, one with URLs and one with links…
Founder here, I suggest you start by not implementing a crawler but use commoncrawl.org instead. The problem with starting a web crawler is you will need a lot of money and almost all big websites are behind cloudflare…
The index we are running right now are all URLs in commoncrawl from 2021 but only URLs with direct links to them. This is mostly because we would need more servers to index more URLs and that would increase the cost. It…
Hello, My name is Josef Cullhed. I am the programmer of alexandria.org and one of two founders. We want to build an open source and non profit search engine and right now we are developing in our spare time and are…
Interesting, in Sweden I only got this story when I made the same searches: https://imgur.com/a/k1Avbtm