HN Explorer

Show HN: Newser, utility written in go to generate a pdf with news content

47 points by lnenad ↗ HN
I've gotten myself a Supernote A5X (awesome device btw) and since it doesn't have a web browser or anything I've wanted to have a way to read news on it. I've hacked together this utility in a couple of days and it works wonders for me personally so I thought it might be interesting to others. It can also be used as a noise free newspaper generator as it removes images/ads/links and other noisy stuff.

https://github.com/lnenad/newser

(there is a screenshot of the first page of the generated pdf)

It scrapes (news) websites for content and puts it into a pdf. For me the pdf location is my dropbox supernote directory so my setup is to run this thing daily and have a fresh pdf with news whenever I want it.

It's rough around the edges probably (currently added crawl support for verge, ars, engadget) but I think it's a good base so if anyone wants to contribute feel free. Some of the stuff I want to add is pictures (maybe), maybe parse the text html to include font styling and other stuff.

I've tried to generalize it as much as possible so the crawling is pretty much automatic and is controlled by a config file where you define "rules" on how to parse the website.

18 comments

[ 2.9 ms ] story [ 50.1 ms ] thread
enw
Cool idea! With the user-hostile modern web, there's something appealing to having a visibly finite, static, and focused piece of content.
lnenad
Thanks, here's how it looks on my device

https://i.imgur.com/ZlfwKQj.jpg

TheFreim
Is the amount of words per line / margins controlled by the config? Configuration was a bit messy (to me) so I figured I'd ask. If I used this I know I'd like to have large margins and less words per line since it's easier to read and mark up.
lnenad
Not for now, but I plan on adding more styling customizations, margins are pretty simple to add so I'll do it probably tomorrow. I also plan on doing some more complex stuff when I get to it (maybe column layout for two articles per page when the article words is < limit words)
alessioalex
A bit off-topic, but I'm thinking of buying a Supernote A5X myself. What do you like most about it, any tips and tricks? Tnx
lnenad
Pretty on topic considering I built this because of the device lol.

I love it, with the latest update extremely low latency pen, excellent battery life, helped me finally to start being disciplined about notes (couldn't do it with just notebooks), pretty. I also like to draw so it scratches that itch. IMHO the downsides (other than no web browser) are pretty much the regular eink stuff - no backlight, fairly slow UI, some artifacts until you refresh the screen etc... But I like it, imho it's worth the money, and the Supernote team is excellent and responsive to user feedback, which is what I want my money to support in the day and age of closed vaults of idiotic products.

TheFreim
> I've gotten myself a Supernote A5X (awesome device btw) and since it doesn't have a web browser or anything I've wanted to have a way to read news on it.

I've ordered one of these, consistently have seen good reviews. I'll keep this software in mind since it might come in handy for automating some blog feeds I might want to read.

What was the reason you originally got the device? For a lot of people it seems to be a way to "disconnect", gaining the power of traditional note taking/reading/writing combined with indexability and less distractions. Reading news seems like it could just turn it into another gizmo. What do you think?

lnenad
For me it was the fact that I wanted to manage myself better and be more organized/structured. I couldn't do this with notebooks as there is a lack of "something" for me that doesn't quite allow me to be as disciplined as I am with the device right now. Templates are awesome, you get 50 different notebooks by just using a different template, and you can create your own in a couple of minutes (a png image with preset dimensions).

Regarding it being another gizmo by adding news, I think that since I didn't get it for the "disconnection" it doesn't feel like corrupting it. Especially in this format (clean, no images, just information at glance). At least imho.

TheFreim
> I couldn't do this with notebooks as there is a lack of "something" for me that doesn't quite allow me to be as disciplined as I am with the device right now.

I understand this. I prefer notebooks for certain tasks and have some good systems I can use to organize it but having a centralized device is very powerful for maintaing what you call discipline.

> Regarding it being another gizmo by adding news, I think that since I didn't get it for the "disconnection" it doesn't feel like corrupting it. Especially in this format (clean, no images, just information at glance).

Yeah, part of it is the philosophy you bring to it (it's a tool, after all). Also depends on your view of "the news". I have a generally negative view, avoid most of it, so that leads me to having a different outlook.

lnenad
> having a centralized device is very powerful for maintaing what you call discipline.

You hit the nail on the head with this. I've got a searchable database of all of my notes and suddenly it feels much less like a waste of time just dumping random stuff into a piece of paper.

> I have a generally negative view, avoid most of it, so that leads me to having a different outlook.

You and me both, but it keeps the mind occupied sometimes and I love reading about new gadgets. World news, economy et cetera not so much.

k1m
This is great!

If it's useful, I work on a project where we maintain a repository of XPath selectors for extracting article content from many different sites: https://github.com/fivefilters/ftr-site-config - they're based on the original public Instapaper rules.

We also have PDF generation, but it's not really for crawling, and wasn't created for reading on a device like the Supernote, more for printing and reading: https://pdf.fivefilters.org/simple-print/

lnenad
Looks cool, I might steal a few configs lol.
marcellus23
Have you considered using RSS rather than scraping? It might simplify some things. Although the downside is not all sites publish full-text content in their feed, so you might have to do some scraping anyway.
lnenad
I have, since not everyone supports is I wanted to do the harder stuff initially.
alan_n
I've not done used the feature in ages, but I'm pretty sure Calibre can do something similar.
xgdgsc
I use Weave.jl to generate daily news html for my self and close friends by a config file with an array of http apis providing data in csv,figure in base64 or markdown text, easy setup with caddy file-server. When I find a datasource like yours, I add it to the config and my daily news gets longer.
adiM
I had a similar setup for creating PDF files from RSS feeds (https://github.com/adityam/rss2kindle). I was simply downloading the webpage, using pandoc to convert HTML to ConTeXt, and typesetting it via ConTeXt (this gave me a lot of control over the formatting and took care of including external images as well). I had a separate script which emailed the PDF to my kindle address.

The script worked reliably for multiple years until I stopped using the kindle. I now have a SuperNote A6X and both pandoc and context have improved significantly in the last decade, so I should give this another shot.