I really like this search engine. Particularly when I want to jump face-first into researching a new topic that I know almost nothing about, the signal-to-noise ratio is often so much better than Google.
Honestly it's not working quite as I want it to right at this moment, it's gotten a bit slow due to a thing I'm testing, and it feels like it's not always finding the best results. Still works decently but still a bit wonkier than usual right now.
Only you, as it's sole (or soul even) developer, could possibly be aware of its current state and result sets and how they may or may not be differing from your ideal vision for it.
For the rest of us, it's a really cool project and I, for one, am glad your efforting it...so thanks!
It's one of my smorgasbord search engines run from a script (which
collectively do so much better than Google or any of the legacy
"big-tech" engines)
Why it matters
One of the great American philosophical orators, Rick Roderick (a
Texan cowboy philosopher and academic twin of Bill Hicks), spoke about
the importance of "the margins":
Rick notices that in history certain people do not get a voice;
Women. The Poor. Old and young people. The "mad" and disabled. Race
minorities within any group. Thinkers more than a decade ahead of
their time.
There's even a thing called "marginal analysis" in hermeneutics - not
the economic concept - the analysis of things that have been left out
of history, and why, and by whom (victors writing and re-writing
history etc). Erasure leaves a trace that tells its own tale.
These things have become the grist of social justice today (with all
it's positive and negative sides). But "injustice" is not really
Roderick's point. It hardly needs mentioning that most people get a
bad deal, don't get heard, and their lives fade into obscure
irrelevance.
What matters is the enormous value and importance of marginal
contributions. These are lost. Almost every great discovery, it turns
out on deeper analysis, was made by an (or often, simultaneously
several) obscure nobody years earlier, who didn't provide a recognised
proof in the right journal, or didn't patent the thing. In Plato and
Shakespeare, Roderick says, "It is always the fool who delivers the
important news". Things don't start at the center, they begin on the
margins and diffuse in until they find legitimacy in an acceptable
voice.
A marginal search engine engine is, almost by definition, acting
counter to the status quo, as the Internet Archive stands in
opposition to fickle entropy of bitrot and ephemeral culture.
drBonkers I think most writers get there by reading a lot and we're
recurrences of our influences. Content wise I've found mixing classics
with modern, deep with superficial, serious with humour, is important.
It balances things like a decent exercise regime.
I recommend Teclis, Wiby, Kagi's "non-commercial" lens (mostly powered by Teclis), and and Gnomit (old results only, helps against the recency bias) to add some serendipity to your searches.
There's two I use. Nothing fancy like collecting and sorting results
from multiple engines. Just a bash sequence.
alias se1="se1.com"
alias se2="se2.com"
www-browser se1 $1; www-browser se2 $1 ...
Arrange them in roughly the order best to worse. If a result isn't
satisfactory I hit q (exit for my choice of browser) and it falls
through to the next search engine.
The other is surf-raw (actually written by everyone's favourite
political prisoner, Mr. Assange) You can change the "elvi" to be what
you want, but imho it doesn't really offer much extra because you still
need to remember the aliases you gave each search engine. A sequence
in an alias file hides that from you so I just type
$ search "keywords about thing"
Of course it falls over when different search engines parse the
urlstring in different ways.
Roderick? There's a lot here [1] and the usual life summary here [2].
Sadly he passed away too soon and recordings of his lectures are
a bit collectable.
Make sure you use the API if you want to query my search engine programmatically, as I do use some bot mitigation strategies for the web interface. If you want to use the API for more than just one-off queries, you can also send me an email and get a key so you get your own rate limit.
While this is all true and all good, “marginalia” is a completely different word for a completely different thing. It refers to handwritten content in the margins of books.
The word is a Latin adjective and just means marginal. It's most frequently used for marginal notes, but I don't see why you couldn't use it in other contexts as well.
Marginalia is a breath of fresh air in an internet that seems to be...disappearing. I actually like it's philosophy of not doing "useful" search queries - it's great just to see what's lost behind Google's wall of Reddit/Wikipedia/SEO Blogspam results. Funnily, after playing with it for a bit my gripe is that there isn't a way to exclude results from vanity domain names which are consistently the lowest quality results.
>I actually like it's philosophy of not doing "useful" search queries
Generalising this a little, it sometimes feels like the world has become all about optimisation and productivity, and the beauty of uselessness has gone missing.
Many of my favourite posts/projects on HN are from people who have set out to build something utterly useless in the name of fun - and sometimes they turn out to be useful after all!
> I put the machine together mostly for a search engine, Because I didn't want an actual rack making noise and heat in my living room, the server is made out of consumer hardware:
Some of that is a bit stale, I'm not using ZFS anymore because it eats too much memory, and the SQL database containing metadata is on an Optane 900P, but it's still largely the same machine.
But yeah, turns out, if you mind what you are doing when you write your software, modern consumer hardware can be pretty good.
I also think a lot of the architecture paradigms that are common today have a very high constant cost, which may be the admittance fee for high scalability, but I don't have anything to scale up to anyway so my problem is how to do the most with fixed hardware.
If I had written this like a cloud application, there is just no way it would run on the same machine. I'd need ten times more hardware. I'd also not be anywhere near finished.
Data locality is slept on in terms of performance.
29 comments
[ 4.3 ms ] story [ 78.5 ms ] threadThankfully, I checked poster name. Grats!
For the rest of us, it's a really cool project and I, for one, am glad your efforting it...so thanks!
Why it matters
One of the great American philosophical orators, Rick Roderick (a Texan cowboy philosopher and academic twin of Bill Hicks), spoke about the importance of "the margins":
Rick notices that in history certain people do not get a voice;
Women. The Poor. Old and young people. The "mad" and disabled. Race minorities within any group. Thinkers more than a decade ahead of their time.
There's even a thing called "marginal analysis" in hermeneutics - not the economic concept - the analysis of things that have been left out of history, and why, and by whom (victors writing and re-writing history etc). Erasure leaves a trace that tells its own tale.
These things have become the grist of social justice today (with all it's positive and negative sides). But "injustice" is not really Roderick's point. It hardly needs mentioning that most people get a bad deal, don't get heard, and their lives fade into obscure irrelevance.
What matters is the enormous value and importance of marginal contributions. These are lost. Almost every great discovery, it turns out on deeper analysis, was made by an (or often, simultaneously several) obscure nobody years earlier, who didn't provide a recognised proof in the right journal, or didn't patent the thing. In Plato and Shakespeare, Roderick says, "It is always the fool who delivers the important news". Things don't start at the center, they begin on the margins and diffuse in until they find legitimacy in an acceptable voice.
A marginal search engine engine is, almost by definition, acting counter to the status quo, as the Internet Archive stands in opposition to fickle entropy of bitrot and ephemeral culture.
Well done on this project and good luck.
I recommend Teclis, Wiby, Kagi's "non-commercial" lens (mostly powered by Teclis), and and Gnomit (old results only, helps against the recency bias) to add some serendipity to your searches.
alias se1="se1.com" alias se2="se2.com" www-browser se1 $1; www-browser se2 $1 ...
Arrange them in roughly the order best to worse. If a result isn't satisfactory I hit q (exit for my choice of browser) and it falls through to the next search engine.
The other is surf-raw (actually written by everyone's favourite political prisoner, Mr. Assange) You can change the "elvi" to be what you want, but imho it doesn't really offer much extra because you still need to remember the aliases you gave each search engine. A sequence in an alias file hides that from you so I just type
$ search "keywords about thing"
Of course it falls over when different search engines parse the urlstring in different ways.
[1] http://rickroderick.org/
[2] https://en.wikipedia.org/wiki/Rick_Roderick
https://api.marginalia.nu/
Show HN: Marginalia – Exploration Mode - https://news.ycombinator.com/item?id=30047455 - Jan 2022 (53 comments)
A search engine that favors text-heavy sites and punishes modern web design - https://news.ycombinator.com/item?id=28550764 - Sept 2021 (717 comments)
Generalising this a little, it sometimes feels like the world has become all about optimisation and productivity, and the beauty of uselessness has gone missing. Many of my favourite posts/projects on HN are from people who have set out to build something utterly useless in the name of fun - and sometimes they turn out to be useful after all!
https://memex.marginalia.nu/projects/gemini-server.gmi
> I put the machine together mostly for a search engine, Because I didn't want an actual rack making noise and heat in my living room, the server is made out of consumer hardware:
> Ryzen 9 3900X
> 128 Gb RAM
> 4x4 Gb IronWolf ZFS
> A bunch of SSDs index lookups
But yeah, turns out, if you mind what you are doing when you write your software, modern consumer hardware can be pretty good.
I also think a lot of the architecture paradigms that are common today have a very high constant cost, which may be the admittance fee for high scalability, but I don't have anything to scale up to anyway so my problem is how to do the most with fixed hardware.
If I had written this like a cloud application, there is just no way it would run on the same machine. I'd need ten times more hardware. I'd also not be anywhere near finished.
Data locality is slept on in terms of performance.