HN Explorer

Show HN: I made a tool to clean and convert any webpage to Markdown (markdowndown.vercel.app)

441 points by asadalt ↗ HN
My partner usually writes substack posts which I then mirror to our website’s blog section.

To automate this, I made a simple tool to scrape the post and clean it so that I can drop it to our blog easily. This might be useful to others as well.

Oh and ofcourse you can instruct GPT to make any final edits :D

106 comments

[ 3.5 ms ] story [ 189 ms ] thread
[deleted]
(comment deleted)
LeonidBugaev
One of the cases when AI not needed. There is very good working algorithm to extract content from the pages, one of implementations: https://github.com/buriy/python-readability
asadalt
oh AI is optional here. I do use readability to clean the html before converting to .md.
haddr
Some years ago I compared those boilerplate removal tools and I remember that jusText was giving me the best results out of the box (tried readability and few other libraries too). I wonder what is the state of the art today?
IanCal
How do you achieve the same things without AI here using that tool?
chrisweekly
"How do you do it without AI" is a question I (sadly) expect to see more often.
IanCal
Feel free to answer then, how do you do the same functions this does with gpt(3/4) without AI?

Edit -

This is an excellent use of it, a free text human input capable of doing things like extracting summaries. It does not seem to be used at all for the basic task of extracting content, but for post filtering.

cactusfrog
I think “copy from a PDF” could be improved with AI. It’s been 30 years and I still get new lines in the middle of sentences when I try to copy from one.
IanCal
That's a great use case, you might be able to do this if you've got a copy and paste on the command line with

https://github.com/simonw/llm

In between. An alias like pdfwtf translating to "paste | llm command | copy"

genewitch
i've long assumed that is a "feature" of PDF akin to DRM. Making copying text from a PDF makes sense from a publisher's standpoint.
hombre_fatal
Meh, it’s just the “how does it work?” question. How content extractors work is interesting and not obvious nor trivial.

And even when you see how readability parser works, AI handles most of the edge cases that content extractors fail on, so they are genuinely superseded by LLMs.

foundzen
how is it compared to mozilla/readability?
asadm
it uses readibility but does some additional stuff like relink images to local paths etc., which I needed
foundzen
I have had challenges with readability. The output is good for blogs but when we try it for other type of content, it misses on important details even when the page is quite text-heavy just like blog.
asadalt
yeah that’s correct. i put a checkbox to disable readability filter if needed…
fbdab103
I was honestly expecting it to be mostly black magic, but it looks like the meat of the project is a bunch of (surely hard won) regexes. Nifty.
nyokodo
> I was … expecting it to be mostly black magic, but … the meat of the project is a bunch of … regexes

Wait, regexes are the epitome of black magic. What do you consider as black magic?

fbdab103
Macros? Any situation where code edits other code?

Sure, I could not write a regex engine, but the language itself can be fine if you keep it to straightfoward stuff. Unlike the famous e-mail parsing regex.

jot
Last time I tried readability it worked well with articles but struggled with other kinds of pages. Took away far more content than I wanted it to.
ChadNauseam
This is awesome. I kind of want a browser extension that does this to every page I read and saves them somewhere.
ElFitz
Wouldn’t that describe Pocket, Readwise Reader, Matter, or another one of those many apps?

Edit: read too fast. Didn’t notice the automatic and systematic aspects.

BHSPitMonkey
Pocket saves the address; I don't think they save the content.
intelkishan
Just checked it, they also save the content.
ZunarJ5
Wallabag + Obsidian + Wallabag Browser Ext. It's a manual trigger but its great.
cratermoon
I've found htmltidy [1] and pandoc html->markdown sufficiently capable.

1 http://www.html-tidy.org/

2 https://pandoc.org/

fbdab103
Never heard of tidy, this looks promising.

I am kind of tempted/horrified to run all of my final templated HTML through this and see if I can spot any lingering malformations. Depending on how structured the corrections are, could make it a test-suite thing.

midzer
"Failed to Convert. Either the URL is invalid or the server is too busy. Please try again later."
julienreszka
I think it was hugged to death
Animats
"Either the URL is invalid or the server is too busy".

Aw, tell the user which it is.

asadm
working on it!
asadm
I think it should be up now
franciscop
I also made one like this a while back, you can extract to markdown, html, text or PDF. I found pages that are the straight tool are very hard to SEO-position since there's not a lot of text/content on the page, even if the tool could be very useful. Feedback welcome:

https://content-parser.com/

These are all "wrappers around readability" AFAIK (including mine), which is the Mozilla project to make sites look clean and I use often.

dazzaji
I’m getting the server overload error but assuming this mostly works I’d use it every day!
asadm
can you try now?
dazzaji
Yes, and it works like a charm! I've used it a few times already today and can see this becoming a very regular part of my workflows. Thank you!
katehikes88
links -dump

elinks -dump

lynx -dump

let me guess, you need more?

kwhitefoot
If the page is rendered using JavaScript then yes you need more.

If there were a version of links, elinks, or lynx that executed JS that would be wonderful.

katehikes88
open the page in a browser

save it to disk C-s

run one of the above commands on the saved file

kwhitefoot
That's so far from being the same as lynx executing JS that I don't understand why you posted it.
[dead] eshoyuan
[dead]
jot
Our tool sadly also fails on this: https://url2text.com/u/KYkpBj

The challenge there is that the content is in an iframe.

If you get the URL used for the iframe you can get the content: https://url2text.com/u/kJWaZY

But that's frustrating as it requires two steps.

We might be able to help you get the content from URLs like these in one step. We have quite a bit of power in the Urlbox API that url2text isn't using.

Drop us an email: support@urlbox.com and we'll see what we can do.

blobcode
This is one of those things that the ever-amazing pandoc (https://pandoc.org/) does very well, on top of supporting virtually every other document format.
rubyn00bie
I second this. Pandoc is up there as one of the most useful tools that exist, that almost no one talks about. It's amazing, easy to use, and works. I regularly see new tools in the space pop-up, but someone would have to have a REALLY unique and compelling feature, or highly optimized use case to get me to use anything else (besides Pandoc).
thangalin
I wrote a series of blog posts about typesetting Markdown using pandoc:

https://dave.autonoma.ca/blog

Eventually, I found pandoc to be a little limiting:

* Awkward to use interpolated variables within prose.

* No real-time preview prior to rendering the final document.

* Limited options for TeX support (e.g., SVG vs. inline; ConTeXt vs. LaTeX).

* Inconsistent syntax for captions and cross-references.

* Requires glue to apply a single YAML metadata source file to multiple documents (e.g., book chapters).

* Does not (reliably) convert straight quotes to curly quotes.

For my purposes, I wanted to convert variable-laden Markdown and R Markdown to text, XHTML, and PDF formats. Eventually I replaced my tool chain of yamlp + pandoc + knitr by writing an integrated FOSS cross-platform desktop editor.

https://keenwrite.com/

KeenWrite uses flexmark-java + Renjin + KeenTeX + KeenQuotes to provide a solution that can replace pandoc + knitr in some situations.

Note how the captions and cross-reference syntax for images, tables, and equations is unified to use a double-colon sigil:

https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/docs/ref...

There's also command-line usage for integrating into build pipelines:

https://gitlab.com/DaveJarvis/KeenWrite/-/blob/main/docs/cmd...

[deleted]
(comment deleted)
jamesstidard
Nice. Turn this into a browser extension and id install it. Feel like id forget about it otherwise
jot
Great idea to offer image downloads and filtering with GPT!

I built a similar tool last year that doesn't have those features: https://url2text.com/

Apologies if the UI is slow - you can see some example output on the homepage.

The API it's built on is Urlbox's website screenshot API which performs far better when used directly. You can request markdown along with JS rendered HTML, metadata and screenshot all in one go: https://urlbox.com/extracting-text

You can even have it all saved directly to your S3-compatible storage: https://urlbox.com/s3

And/or delivered by webhook: https://urlbox.com/webhooks

I've been running over 1 million renders per month using Urlbox's markdown feature for a side project. It's so much better using markdown like this for embeddings and in prompts.

If you want to scrape whole websites like this you might also want to checkout this new tool by dctanner: https://usescraper.com/

jph00
Looks nice, but url2text doesn't seem to have an API, and urlbox doesn't seem to have an option to skip the screenshot if you only want the text. And for just the text, it looks to be really expensive.
jot
Thanks!

Sorry it's not clearer but you can skip the screenshot in the Urlbox API if you want to with:

  curl -X POST \
    https://api.urlbox.io/v1/render/sync \
    -H 'Authorization: Bearer YOUR_URLBOX_SECRET' \
    -H 'Content-Type: application/json' \
    -d '
  {
    "url": "example.com",
    "format": "md"
  }
  '
Here's the result of that: https://renders.urlbox.io/urlbox1/renders/5799274d37a8b4e604...

Sorry the pricing isn't a good fit for you. Urlbox has been running for over 11 years. We're bootstrapped and profitable with a team of 3 (plus a few contractors). We're priced to be sustainable so our customers can depend on us in the long term. We automatically give volume discounts as your usage grows.

old_dev
It looks like, if the website presents a cookie message, the tool just gets stuck on that and does not parse the actual content. As an example, I tried https://www.cnbc.com/ and all it created was a markdown of the cookie message and some legalese around it.
jot
It's not easy working around things like that. But here's how it could work: https://url2text.com/u/wYVake

We were lucky to build this on a mature API that already solves loads of the edge cases around rendering different kinds of pages.

ben_ja_min
I've been looking for this! My method requires too many steps. I look forward to seeing if this improves my results. Thanks!
sigio
Tried it earlier today, but it was hugged to death. Tried it now, but it only gave me the contents of the cookie-wall, not the page I was looking for.

Tried on another page of the same site, then it only gave me the last article on a 6-article page, some weird things going on.

brokenmachine
I use markdownload extension for Firefox. Seems to work pretty ok.
radicalriddler
Is this open sourced anywhere by any chance? Are you using GPT to do the conversion, or just doing it yourself by ways of HTML -> Markdown substitutions?
selfie
Vercel!! Watch out for you bill now this is being hugged. Hopefully you are not using <Image /> like they pester you to do.
asadalt
it has actually scaled pretty well and has been negligible cost-wise.

I didn’t have to do anything to handle HN traffic. Just a boilerplate nextjs app.

compootr
it's pretty sad, they motivate you all throughout their docs to use it, when just a webp would work ok

it's literally that meme of the bus — the happy guy is you and vercel and the sad one is your wallet

(unless you need dynamic scaling and minification though)

aleksandern08
What’s the problem with <Image />?
asadm
Vercel charges for image optimizations/dynamic scaling. People get wrong perception that it should be a free service because the DX is so easy...