2markdown – Transform Websites into Markdown (2markdown.com)

> empower your AI models

pardon?

pandoc as a service, mad times we live in, best of luck I suppose

Should definitely use a try 1st one free without login demo (maybe with a strong captcha if missuse costs are an issue)

Why not just use something like https://github.com/mozilla/readability

And not pay $0.01 per request?

There’s a node version too https://www.npmjs.com/package/@mozilla/readability

I built a similar thing in python using Playwright and Pandoc [0]. It's used by aider's `/web <url>` command that lets you paste a markdown version of any webpage into your AI coding chat. This helps if you want to include docs for an obscure or non-public package/api/etc with the LLM while coding.

I really value dependencies which are easy for all users to install, cross-platform. Playwright is nice because it has a simple way to install its dependencies on most platforms. And the `pypandoc` module provides a seamless install of pandoc across platforms.

The result turns most web pages into nice markdown, without requiring users to solve some painful platform specific chromium dependency nightmare.

For example, it produced this markdown version [1] of the pyinfra getting started doc [2].

[0] https://github.com/paul-gauthier/aider/blob/main/aider/scrap...

[1] https://gist.github.com/paul-gauthier/6c8dd16a6bceb9fd287494...

[2] https://docs.pyinfra.com/en/2.x/getting-started.html

I'll stick with using Markdownload (https://github.com/deathau/markdownload) when I need to do something like this.

Not interesting in the service but fascinated by this new wave of re-modularization and re-integration of services through the lens of Ben Thompson aggregation theory.

Modularizers like 2markdown are building single purpose services optimized for LLM workloads

Aggregators like ChatGPT fights for end-to-end AI experience.

It's just exciting to watch.

> From this

> This content displays when the browser does not support custom elements.

> To this

> This content displays when the browser does not support custom elements.

this happens with ublock origin on

These two blob posts by the author seems to be revealing:

How to Build a Startup Without Funding https://andinfinity.eu/post/2024-04-02-how-to-build-a-startu...

What's the minimal set of user stories for a Micro-SaaS MVP? https://andinfinity.eu/post/2022-06-06-whats-part-of-an-mvp-...

It might be helpful when accessing news websites outdoor (since my internet provider rate limits hard). It’s basically a proxy, right?

I’m not even going to try though because there’s a signup wall.

If you’re looking for an open core version of this check out firecrawl.dev

I have a wrapper for the many ways you might convert something to a text document in the terminal.

https://github.com/reagle/pandoc-wrappers/blob/main/doc2txt....

I used to be intrested in converting/saving websites to my own archives. The web is rotting away and saving a local copy is the easiest way to make sure I keep my references.

This, this ain't it chief. If you're looking for local saving, use firefox extensions. MarkDownload and SinglePage are both very good for backing up single web pages.

I have been using lynx to integrate a chatbot with web content. Here's the Python function that converts a website to plain text:

  def website_to_text(url):
       output = subprocess.check_output(["lynx", "-dump", "-nolist", url], stderr=subprocess.PIPE, universal_newlines=True)
       return output.strip()

While a hosted service can be convenient, I find that using a text-based browser like Lynx gives almost the same results and consistent results.

Wow, "pay as you go html to markdown" seems like a very hard sell in this year of our lord 2024. I've got a brace of converters for various parts of this problem, some of them are pushing a decade in age. You have products like Zapier or the open source ActivePieces to restrain extraction to specific elements. Finally, several pre-wrapped AI models for this specific purpose.