Show HN: Crul – Query Any Webpage or API (crul.com)
With crul you can crawl and transform web pages into csv tables, explore and dynamically query APIs, filter and organize data, and push data sets to third party data lakes and analytics tools. Here’s a demo video, we’ve been told Nic sounds like John Mayer (lol) (https://www.crul.com/demo-video)
We’ve personally struggled wrangling data from the web using puppeteer/playwright/selenium, jq or cobbling together python scripts, client libraries, and schedulers to consume APIs. The reality is that shit is hard, doesn’t scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues. The tools we love to hate.
Crul’s value prop is simple: Query any Webpage or API for free.
At its core, crul is based on the foundational linked nature of Web/API content. It consists of a purpose built map/expand/reduce engine for hierarchical Web/API content (kind of like postman but with a membership to Gold's Gym) with a familiar parser expression grammar that naturally gets the job done (and layered caching to make it quick to fix when it doesn’t on the first try). There’s a boatload of other features like domain policies, scheduler, checkpoints, templates, REST API, Web UI, vault, OAuth for third parties and 20+ stores to send your data to.
Our goal is to open source crul as time and resources permit. At the end of the day it’s just the two of us trying to figure things out as we go! We’re just getting started.
Crul is one bad mother#^@%*& and the web is finally yours!
Download crul for free as a Mac OS desktop application or as a Docker image (https://www.crul.com) and let us know if you love it or hate it. (https://forms.gle/5BXb5bLC1D5QG7i99) And come say hello to us on our slack channel - we’re a friendly bunch! (https://crulinc.slack.com/)
Nic and Carl (https://www.crul.com/early-days)
80 comments
[ 3.4 ms ] story [ 168 ms ] threadThere's a short writeup of this pattern here: https://www.crul.com/docs/examples/how-to-find-filters
Thanks for checking it out!
It would take some thinking and planning, and it's possibly not even a good idea ;) But generally any "data source" is packageable as an FDW as long as you can model it in such a way that you can reasonably implement certain functions for operations like table scans. For most FDWs, this is easy and the tradeoff of a large query is usually limited to excess bandwidth and latency while the query executor reads the result from the FDW. But with a live source pointing to a crawler instance, a table scan could in the worst case mean waiting for the crawler to parse the responses to hundreds of rate-limited network requests. So it's probably better to ingest the data once (and/or periodically) for a particular crul "table" (whatever you decide that means) rather than to query it live.
Fortunately, you can still write an FDW as the adapter layer, because Splitgraph ingests data on a schedule by querying the FDW of the live data source (while tolerating a long-running query). Alternatively (or additionally) you could write an Airbyte adapter which we also support, but only for ingestion - if you want live queryable tables then an FDW is necessary.
We've been interested in adding something like this (think Apify + Postgres) for a while. If done well it could be really cool. Let me know if you want to talk about it: miles@splitgraph.com
I just got to the section on destinations ("Stores"). Very cool. If you're building an Enterprise plan where you manage the infrastructure for your customers, we can deploy a dedicated white-labeled deployment of Splitgraph Cloud to any of the three major clouds. Perhaps you could find a use for that in your backend infrastructure.
ETA: Also, if you can write to Postgres, you can write to the Splitgraph DDN [0] with most DML and DDL statements, including INSERT, and CREATE TABLE. So even without any FDW, you might be able to add Splitgraph as a "destination" for your users (who would for the most part just need to provide an API keypair).
[0] https://www.splitgraph.com/docs/add-data/from-ddn
Edit: just saw "airbyte" in a bunch of places, which I assume answers my question. So updated question: airbyte works well for ya?
Yes, content changes, but so does structure. If I'm interested in content that shows up in a news feed div, and that div is renamed or moved as part of a site redesign, what happens?
I've worked on a bunch of tools in the past that do similar things, and structural changes were the kryptonite for all of them.
A secondary problem is when you use particular content as a reference point, and that content is later updated. Now your reference point is gone!
We have done some preliminary work in some AI or other intelligence for pattern recognition to be able to handle structural changes better, but still have lots of work.
But the expanding and querying concepts also make a lot of sense with APIs, which tend to be a little more stable.
The second hardest part was 30% of our clients all used the same hosting provider, which would start to fail at 10-20 req/s. We had to throttle the sites by IP, cluster-wide.
Crul sort of operates as a text only browser when interacting with a single page at a time, but when you expand and open up multiple tabs it becomes a little more challenging. We have the concept of domain policies which allow you to control how quickly/slowly you access something. There are also some puppeteer level options that could be relevant, even a headful toggle.
We have not invested too much time into this yet as we focused on getting the core functionality working. We think there are use cases (particularly with APIs) that don't run into this problem, but if it comes up more often we'll come up with some options.
The only kind of emulation where I have seen success is mobile and in that case you need to run a device emulator.
edit: I have quite a bit of experience with Akamai and other vendors =)
We've been thinking of ways to make the pronunciation a bit clearer, maybe a mascot or something. Open to ideas!
What are some of your experiences with async networking, btw? Did you find that it didn't really handle concurrency well in practice? Or were there other issues? I've written a lot of async networking code but always found it horrible to profile.
We've built using a microservice architecture to allow us to scale out the parts that need to scale, mainly the workers, which interact with a queue, although we'll need to move from the parts that are currently nodejs for some perf gains and a smaller footprint. All those microservices are consolidated for the desktop variants.
Network concurrency is mostly throttled by our domain policy manager (named "gonogo" - lol) at 1 req/per outbound domain a sec. It's a little slow for a default, but also configurable and provides a nice guardrail for api request limits, etc. Overall async networking has been quite tricky, esp with retries, etc., and we're still iterating on it. Agreed on the profiling difficulties.
Why was it so slow? Is there a default delay between requests or something?
We found that it becomes too easy to break API request limits or spam a website otherwise.
However if you rerun that query it should load pretty instantly due to the caching layers, so the actual querying/filtering of the data part is smoother/faster.
Every time I see that, the "2 hardest things" springs to mind. Is there a clear-caches option, or I guess the opposite question: does that process honor the HTTP caching semantics? Scrapy actually has a bunch of configurable knobs for that (use RFC2616 Policy ( https://docs.scrapy.org/en/2.8/topics/downloader-middleware.... ), write your own policy, or a ton of other stuff: https://docs.scrapy.org/en/2.8/topics/downloader-middleware.... )
Your provided links are interesting and something for us think about some more. Honestly, I would be quite interested in hearing more about your experiences.
Wait, so we literally can't go faster than 1 req/s unless we pay?
I have to say I'm pretty disappointed :/
Scheduling and Domain Policies were the main features we chose to gate initially as they don't affect core functionality other than performance and deployment.
Don't spam APIs. That said, if you're determined to do so, there's not much this or any other tool can do to stop you from trying.
Crul is a really easy way of populating tools that need data, whether it's just a one time thing for a demo/static data set or a scheduled data feed, so this kind of usage makes sense to us.
We're taking a different apporach to our enterprisey competitiors. ;0)
>we currently have the scheduler set as an enterprise feature.
Understandable.
Here's a query that shows an option, but the table command will be far more straightforward.
open https://www.w3schools.com/html/html_tables.asp --dimension || filter "(nodeName == 'TD')" || groupBy boundingClientRect.top || table _group.0.innerText _group.1.innerText _group.2.innerText
"The reality is that shit is hard, doesn't scale (classic blocking for-loop or async saturation), and comes with thorny maintenance/security issues."
Every web user's needs are different. One person might have a task that they struggle to accomplish while another might have one that presents no major challenges. As a web user, I transform web pages to CSV or SQL. I log HTTP and network requests. I do this for free using open source software. No web browser needed. No docker image needed. Works on both Linux and BSD.
For me, the web is a dataset from which I retrieve data/information. "Tech" companies want to the web to be more like a video game, with visuals and constant interactivity.
Related to the quote, we've seen interest for API data wrangling, where prepackaged data feeds can be cumbersome to edit, or other implementation details become challenging like credential management, domain throttling, scheduling, checkpointing, export, etc.
It's also interesting for webpage data when you need to use a particular page as an index of links to filter and crawl. We've tried to build an abstraction layer around that.
Initially, we were mainly focused on webpages, and wanted to bypass the visuals of the browser and use a headless browser to fulfill network requests, render js, etc. then convert the page to flat table of enriched elements. With the APIs as another data set there is some work for us to do around language.
We're now trying to figure out what workflows are most relevant for crul to optimize around, as well, honestly - we just built what we thought was cool. Some features/workflows will certainly be more straightforward with existing tools and software - especially for a technically savvy user.
Taking a quick look at the Amazon site these product pages appear to be enormous in size. Interestingly, the website requires a "viewport-width" header. Otherwise one gets directed to a CAPTCHA.
The product page I checked already has some structered data in the form of JSON, including keys such as
Thus, CSV with product name, price and ASIN would appear to be easy. No need to mess with the HTML.Other data such as, e.g., delivery time, seller, where the item ships from and number left in stck can be extracted from the HTML.
Delivery time is in a <span> that contains "data-csa-c-delivery-time".
Seller, shipping info and number left in stock are under a <span> with class="a-size-base _p13n-desktop-sims-fbt_fbt-desktop_shipping-info-show-box__17yWM"
One needs to decide what data one wants from the page.
The way to present an example on which to evaluate a "new" solution such as the one in this thread is to present a problem, e.g.,
Get data items x, y and z from website xyzexample.com.
In the majorty of cases I see submitted to HN, it is impossible to benchmark these "new" solutions against existing ones because no example websites are ever provided.
I can't tell if the message is from Brave or from the ISP though.
https://github.com/crul-oss
open https://news.ycombinator.com/item?id=34970917 || filter "(attributes.class == 'toptext')" || table innerText
open https://news.ycombinator.com/item?id=34970917 || filter "(nodeName == 'BODY')" || table innerText
Often you can use crul to discover the html structure in the results table. With a `find "text string"` to filter rows and then a filter on the column values that identify the desired elements.
Screenshot: https://www.dropbox.com/s/i254xxbdenmqde1/screenshot%202023-...
Have you thought about open sourcing it? I realize you want to sustain a business, but an open core model with commercial features would allow you to do that. A free tool like this would be a welcome addition to the ecosystem, most of which is FOSS already, and would help build a community around your product.
Open to any perspectives on this.
1. Many JSON endpoints (especially not meant for public consumption) return a somewhat deeply nested list, and somewhere in that list are the individual items. Can your tool speed up (or automagically solve) getting just the itemns? I found that it just gives me thousands of columns (and truncates the results), where I would have wanted, say, two thousand rows of 23 columns. I couldn't wrangle the JSON within crul.
2. Many smaller sites still use wordpress, and may manually lay out items in a visual hierarchy that isn't immediately obvious in the structure of the HTML. Then you have to go in and parse every "row" and every "column" using xpath or css selectors. Crul wasn't particularly helpful for this either.
3. A scraping workflow requiring authentication, post request for searching, get request for picking the correct result, post request for downloading pdf and parsing said pdf... well, I couldn't get this to work at all.
Hope you'll end up solving such problems automagically and I end up paying you for it.
1. Although it has limitations, were you able to try the normalize command? https://www.crul.com/docs/queryconcepts/api-normalization
2. Will need to think about this some more.
3. Although we don't yet handle pdfs, the rest of the flow is one we aiming to accomplish with crul. Other than pdf, the pieces should be there and would love to understand this further
2: Probably too niche to be worth it for you :)
3: Parsing pdfs automagically isn't easy, but handling downloads and images and storing them in a bucket would go quite far (I guess there's a way to do that, but I didn't immediately see a "simple" example).
It sounds like it's potentially a great tool, the only open question is really, is it worth studying the docs and implementing a process in crul as opposed to whatever language I'm familiar with already?
Asking all those as we are working on CloudQuery (https://github.com/cloudquery/cloudquery) so been dealing with a lot of those underlying challenges as well.