Not the same but this gives me an idea… what if there was a map reduce for doms as a web primitive. Like imagine if I could make a dom (or feed) that was some selection and transformation of another dom
I made a CGI program that ran CSS selectors against URLs and returned the output. I debated making it public and then realized I probably didn't want to run an open proxy. I'm curious how long this will last.
The few times I actually tried it, it worked badly, with huge chunks of text content missing from the page. Makes me wonder if with modern web the task has became so difficult even a browser couldn't pull it off, or if they just wasn't trying to do a good job with the feature.
It takes an array with the entries as input, not a web page. But I guess the HTML parsing should take no more than another few lines? For HTML parsing, I have good experiences with the lxml module which is in the Debian repos. It is fast and works pretty well.
I wrote a similar thing in go (using chromedriver, so it could handle things that need JS).
Handled most things nicely, but I found a few sites where I wanted multiple selections to be combined into one document.
I emailed the result to myself, turning any images into attachments; this meant my “feed reader” had read/unread tracking that synced across devices, some html support, folders, offline viewing, etc.
15 comments
[ 3.4 ms ] story [ 31.4 ms ] threadThe bad news: so did the 503 page.
And the GitHub url (hopefully easy to host your own instance): https://github.com/kevinschaul/feedmaker
Has anyone tested to see if it works with Blogtrottr which will email you whenever there's a new item in an RSS feed?
Just since this doesn't seem like it even includes a date field in the RSS? And of course no guid. So I'm wondering how compatible it winds up being.
The few times I actually tried it, it worked badly, with huge chunks of text content missing from the page. Makes me wonder if with modern web the task has became so difficult even a browser couldn't pull it off, or if they just wasn't trying to do a good job with the feature.
59 requirements, including Django, seems pretty heavy though?
For my own RSS feed, I use this 48 line Python file with no dependencies outside the standard library:
https://github.com/no-gravity/atomfeed.py
It takes an array with the entries as input, not a web page. But I guess the HTML parsing should take no more than another few lines? For HTML parsing, I have good experiences with the lxml module which is in the Debian repos. It is fast and works pretty well.
Handled most things nicely, but I found a few sites where I wanted multiple selections to be combined into one document.
I emailed the result to myself, turning any images into attachments; this meant my “feed reader” had read/unread tracking that synced across devices, some html support, folders, offline viewing, etc.