Thanks for posting that, it is exactly what I need.
I want to scrape a diverse selection of web pages for 'the gist' of the main content purely to put into a solr search engine. At the moment I am doing surprisingly well just going on the meta descriptions - in essence meta descriptions should be what I am after - but meaningfully scraped content is the ideal and what I need if no meta description available.
I am sure you have lots of grand plans for this, however, one thing that I wish I could do is search through my web history, at a content level.
For instance, earlier today I found some code was no longer working properly. I could remember bits of the tutorial I needed first time around but finding it again was a chore (on a narrow subject then every search result from Google is purple/visited, so you can't immediately identify the link).
If I could have done some 'search my previous browsing history' then I could have got there a lot quicker. In this 'imaginary search' only the pages that are kept open and scrolled through get 'internally archived'. There could also be snapshots of the pages in question in the results as an aid to memory.
Another thing I would like to see done with this type of tool is a sensible spell check. For decades we have had spell checks in text entry areas but never on a finished web page. A little widget to show the words with problem spellings with wavy underlining that would work on any web page would save no end of woes for people that have to proof read things online. If it also highlighted sentences too long for good copy that would be very useful too.
Can these sorts of applications be built on what you have here?
Thank you for this, I just started using node-readability for a project and I'll gladly give this a spin. The second url [0] I tried passing it however unfortunately returned an empty body. node-readability is able to parse the contents of tagesschau.de seemingly without problems.
How do you deal with web pages, where the content is dynamically created using javascript/ajax? Do you execute this JS, or just ignore this growing antipattern?
Send email message to read@snipek.com , with a URL in subject line. Then you'll get a readable email of that web page. You can read it offline on mobile mail client. That's how I read long web articles these days.
If you get used to this way, using email to read web articles, you can use it to navigate web pages in mailbox. For example, send anything to hn@snipek.com to get hack news, and click & send mail to read each articles.
Good to see an OSS version. How does it handle pagination? From my experience (Pocket, Readability, Readable (now Evernote Clearly), Boilerpipe) that's the big differentiator. But frankly, with the increased use of media queries, I find myself using this view less and less, and the only feature I miss is pagination.
The optimal reading environment on a responsive website is usually the tablet view. Narrow enough to make jumping lines easy, narrow enough to prevent big ads on the side columns. I'd love to see a tool that could force media queries on bigger displays and simply center the text. You'd still get a feel for what the web designer was going for, but it's much cleaner. On none responsive sites, it could use the same metrics as this tool to determine which elements are article elements and simply center them and remove the rest, while maintaining the fonts, colors... The medium is the message and all that jazz.
Very simple request for the developer, you couldn't put up screenshots/examples of text that has been extracted along with a (cached) link so that people can see how it looks without downloading and testing it?
23 comments
[ 4.0 ms ] story [ 62.5 ms ] threadI want to scrape a diverse selection of web pages for 'the gist' of the main content purely to put into a solr search engine. At the moment I am doing surprisingly well just going on the meta descriptions - in essence meta descriptions should be what I am after - but meaningfully scraped content is the ideal and what I need if no meta description available.
I am sure you have lots of grand plans for this, however, one thing that I wish I could do is search through my web history, at a content level.
For instance, earlier today I found some code was no longer working properly. I could remember bits of the tutorial I needed first time around but finding it again was a chore (on a narrow subject then every search result from Google is purple/visited, so you can't immediately identify the link).
If I could have done some 'search my previous browsing history' then I could have got there a lot quicker. In this 'imaginary search' only the pages that are kept open and scrolled through get 'internally archived'. There could also be snapshots of the pages in question in the results as an aid to memory.
Another thing I would like to see done with this type of tool is a sensible spell check. For decades we have had spell checks in text entry areas but never on a finished web page. A little widget to show the words with problem spellings with wavy underlining that would work on any web page would save no end of woes for people that have to proof read things online. If it also highlighted sentences too long for good copy that would be very useful too.
Can these sorts of applications be built on what you have here?
P.S. Funny that you wrote this 186 days ago:
https://news.ycombinator.com/item?id=6581317
Times have changed...
[0] http://www.tagesschau.de/ausland/krise-in-der-ostukraine112....
How do you deal with web pages, where the content is dynamically created using javascript/ajax? Do you execute this JS, or just ignore this growing antipattern?
Unfortunately they don't seem to have a desktop equivalent, yet.
And here are the mock-ups: http://people.mozilla.org/~mmaslaney/readermode/index.html
The optimal reading environment on a responsive website is usually the tablet view. Narrow enough to make jumping lines easy, narrow enough to prevent big ads on the side columns. I'd love to see a tool that could force media queries on bigger displays and simply center the text. You'd still get a feel for what the web designer was going for, but it's much cleaner. On none responsive sites, it could use the same metrics as this tool to determine which elements are article elements and simply center them and remove the rest, while maintaining the fonts, colors... The medium is the message and all that jazz.