It depends on the use case. Say you want to get all images on a page you'd just use `img`. If you are going to find JSON-LD blocks defined in a set of pages, you can use - `script[type='application/ld+json']`.
For MVP, I wanted to give the option of free-form selectors and then offer more targeted queries based on what people use.
Consider offering responses other formats, possibly protobuf. Then you can sell it as a solution for limited IoT devices where a full HTML parser is too much. If your service avoids having to host a custom server, it could be very useful.
Looks interesting. I wonder what kind of market this app might serve. For larger apps, I would worry about support. 5 dollars per year tells me that the developer is doing this as a hobby. For small side projects, I can see tinkerers building this themselves.
It's actually a utility I built for my own use (I bootstrap Pragma full-time). I thought there might be an interest among other developers for a similar utility. That's why I thought to put it out (wanted to charge $5 so I can reduce abuse).
If you are interested in using it for a larger app and concerned about support, please drop me an email (see profile). I can set you up an instance with better rate limits and service gurantees.
I would make the 5$ price and token validity much larger, like "4rem" or something, I was looking at the CC input field and thinking "seriously? how much will you charge?"
Very neat. But the question is, how they pay the cost of the server? I mean, do you think a web service without any API keys, subscriptions, etc. is reliable?
I'm shocked to hear more and more people running an HTTP api from a $5 DO instance. It's enticing enough for me to learn, but I want to know, what makes golang so scalable?
I love the idea of just serving a production HTTP API from a single go file.
It's not DO. It's scaleway C1 arm server, even cheaper.
Golang produces native code with very good runtime with green threads which makes it very easy to do async programming and write efficient network code. But language won't save you from inefficient algorithms and dumb code of course.
> I love the idea of just serving a production HTTP API from a single go file.
You may have mistaken by single Go file in my service but it is actually a lot of code included spread across several libraries.
In order to write this service I had to write my own open graph library, oembed library, html parsing library etc..
If you are interested the same is possible with https://link.fish/hn-early-access/ which has a Zapier integration. It allows to send an URL and it then returns its data.
However, you would have to send me an email to info@link.fish because the integration is still in beta mode and so only available after invitation.
How do you handle sites that have scraper prevention? Such as captcha, IP throttling, etc.
I'd pay $100-$500 per-month for a service that could reliably scrape some particularly difficult sites. With that being said, I'd need the service to be able to handle ~100 req/s in bursts and 2-4 req/s on average.
32 comments
[ 2.4 ms ] story [ 72.3 ms ] threadFor MVP, I wanted to give the option of free-form selectors and then offer more targeted queries based on what people use.
If you are interested in using it for a larger app and concerned about support, please drop me an email (see profile). I can set you up an instance with better rate limits and service gurantees.
There are limits set for access token too. I actually rushed to ship before putting out a TOS which mentions limits.
General example: http://api.proc.link/oembed?url=http%3A%2F%2Fpage.rest
Youtube: http://api.proc.link/oembed?url=https%3A%2F%2Fwww.youtube.co...
Facebook: http://api.proc.link/oembed?url=https%3A%2F%2Fwww.facebook.c...
Regarding reliability: all the code is opensourced and if you want to use the service in production - you can run this on your own server.
It is running for more than 2 years now, never failed :)
Golang is pretty stable.
Also the server is behind Cloudflare which caches responses and protects from DDoS, this can be another reason for low RPS to the server.
I love the idea of just serving a production HTTP API from a single go file.
Golang produces native code with very good runtime with green threads which makes it very easy to do async programming and write efficient network code. But language won't save you from inefficient algorithms and dumb code of course.
> I love the idea of just serving a production HTTP API from a single go file.
You may have mistaken by single Go file in my service but it is actually a lot of code included spread across several libraries. In order to write this service I had to write my own open graph library, oembed library, html parsing library etc..
I wanted to make an analog of http://embed.ly/code but lost interest in the middle.
I can code difficult logic and all the highload server stuff but when it comes to front end, billing, images etc.. I'm getting bored too quickly.
If someone wants to help me make a business out of this: mdxytop at gmail.com
https://github.com/keyle/json-anything
I'd pay $100-$500 per-month for a service that could reliably scrape some particularly difficult sites. With that being said, I'd need the service to be able to handle ~100 req/s in bursts and 2-4 req/s on average.