We don't support logging in yet, but it's a feature we're working on adding. Scripting will also be cool, but it's right now further down our feature queue
I thought it was intentional. Swedish chef style. I like it, but I'll need to go back and re-read to understand how I can use this on other pages than the homepage/demo page. I've nothing immediate to try it with right now. :)
Edit: I'll watch the video after work, probably will clear everything up for me.
I'm normally a bit worried when a thread quickly fills up with praise, but this looks very nice.
It's something I have thought about, as I'm sure many people who have done any amount of scraping have, but never went forward and tried to implement. The landing page with video up top and in-line demo is a pretty slick presentation of the solution you came up with. Good job.
Looks awesome, however I keep getting errors and 404s. Could this be an issue on my end (seems to be working for others) or just HN making the servers beg for mercy?
If it changes the format significantly, the scraper will break, so for now you'll have to use the tool to rebuild. You will see on your API status page that it's down. As for robots.txt, we do respect it... for now we're leaving that to the user, but we're trying to implement a proactive way of checking for disallows and stopping those scrapers from being built.
At the moment, we rely on users to be responsible. We spell it out in the terms and FAQ. We've been in private beta, keeping usage very limited until today. We fully understand the seriousness of the issue as we scale. We're committed to becoming a responsible bot that respects robots.txt
I would say how you're scraping differs from say how Google, a search engine, scrapes. I'm not sure there is a way in robots.txt to define for each use? Knowing the data in a structured way, but then allowing it to be displayed in full off-site is quite different than using the scraped data for linking into a website.
Feedback from webmasters is really helpful for us. We want to make sure we're making data available via API responsibly, so would love to hear your suggestions/ thoughts as we define a scalable solution.
Well done on the product & solving a clear need! This is extremely useful for hackathons/prototyping. I also loved the live demo in the blog post and you did a wonderful job with the design/layout/colorscheme of the site.
thanks. glad you like it. pagination, dynamic tabs (and crawling in general) is a big feature we really want to add soon. a lot of people are asking for it. the challenge will be integrating it with the current UEX which we're trying to keep super simple.
Another feature, simple one: Allow to add some filters to the data stream. For example: only posts that contain word "bitcoin" in the name or only those with 50 upvotes or more.
Thanks. We support regex matching now. Try dragging to select text, if there's a relevant regex pattern and kimono will find it (there's an example inthe blog post). You can preview (and soon, you'll be able to edit) the CSS and Regex also in advanced mode.
JS-heavy sites can be tricky. We position it so it should execute after most of the on-page JS, so it handles a lot of cases. There are still sites that break it though.... we're trying to tackle these guys one by one right now, as we try to generalize a broader solution
Great request... it's on our feature shortlist. Definitely a feature we want to implement as soon as we can (after we tackle some basics like pagination and getting images)
Thanks... yes, public data from governments is a great use case. Often a lot of apps built using scrapers will wind up driving up traffic/ sales a the source site so it's okay. We want to do responsible web scraping, so will respect webmasters robots.txt files to make sure it's legal.
I love the execution, but I also see inherent problems.
Robots.txt is just a convention to advise crawlers. I'm confident most sites explicitly state this is against their terms of service.
You will encounter terms along the lines of:
"Unauthorized uses of the Site also include, without limitation, those listed below. You agree not to do any of the following, unless otherwise previously authorized by us in writing:
Use any robot, spider, scraper, other automatic device, or manual process to monitor, copy, or keep a database copy of the content or any portion of the Site."
You've got a valid point. We want to eventually create a space that allows responsible scraping - so webmasters can have access to analytics on what's being scraped and can explicitly turn off kimono APIs for their domains if they see fit. We also think there are use cases for people who own their own data. Often, APIs will provide a way for companies to streamline their internal app development and figure out what to expose to the developer community before investing in an expensive API deployment.
The law isn't entirely blind to conventions, though. They don't guarantee anything, but if a court understood that there exists a convention for saying "no robots, please", and the robot operator in question followed it, then a court could well look less favorably on the damages claims of a website operator who didn't make use of the widely known convention.
I'm not sure scraping itself is illegal, depending on what you're doing with the data. (Though it may be against a site's Terms of Use which may be binding. IANAL.)
I can tell you that on several occasions I've scraped commercial sites with the permission of the owner. They want me to have access to the data but don't have the time or ability to create a proper API.
I've written more web scraping code than I care to admit. A lot of the apps that ran on chumby devices used scraping to get their data (usually(!) with the consent of the website being scraped) since the device wasn't capable of rendering html (it eventually did get a port of Qt/WebKit, but that was right before it died and it wasn't well integrated with the rest of the chumby app ecosystem).
This service looks great, good work! But since you seem to host the APIs created how do you plan to get around the centralized access issues? Like on the chumby we had to do a lot of web scraping on the device itself (even though doing string processing operations needed for scraping required a lot of hoop jumping optimization to run well in ActionScript 2 on a slow ARMv5 chip with 64mb total RAM) to avoid all the requests coming from the same set of chumby-server IP addresses, because companies tend to notice lots of requests coming from the same server block really quick and will often rate limit the hell out of you, which could result in a situation where one heavy-usage scraper destroys access for every other client trying to scrape from that same source.
Access, legality and rate limiting issues come up a lot. We're working on a couple things to address them. The first is an intelligent job distribution system that consolidates scrapes across users and hits sites (and pages) at human-like intervals. the second is to create a portal for webmasters that allows them special privileged access to analytics on data being extracted from their sites, and the ability to "turn on or off" kimono APIs if they see fit. this way, via kimono, a webmaster at chumby could "provision" certain kimono users. we're still yet to see whether the later works out. thanks for the input
This looks really useful, and I'm trying to figure out if I could use it on a project I'm working on, but hitting an issue. I sent a support message. Nice job!
This is a great tool! In a past life we needed a web scraper to pull single game ticket prices from NBA, MLB, and NHL team pages (e.g. http://www.nba.com/warriors/tickets/single). We needed the data. But, when you factor in dynamic pricing and frequent page changes you are left with a real headache. I wish Kimono was around when we were working on that project.
I love how you can actually use their "web scraper for anyone" on the blog post. Very cool!
HTTPS is definitely a problem for proxy servers unless you the proxy server rewrites all the URLs in the html pages loaded as well as all the URLs of the Ajax calls to point back to the proxy server.
Nice work, this is much better than I expected! Does it require Chrome? It doesn't seem to work in Safari for me. Also, does Kimono work for scraping multiple pages or anything that requires authentication?
7.0.1, the latest. I also don't have Flash installed, but it doesn't look like you're using Flash. The entire top bar doesn't show for me. Feel free to email me and I can send you screenshots.
233 comments
[ 3.2 ms ] story [ 232 ms ] threadMore web apps need an undo button.
Oh, typo: "Notice that toolbar at the toop of the screen?"
Edit: I'll watch the video after work, probably will clear everything up for me.
Looks pretty good, but it does not really replace my scrappers. Maybe some of them...
It's something I have thought about, as I'm sure many people who have done any amount of scraping have, but never went forward and tried to implement. The landing page with video up top and in-line demo is a pretty slick presentation of the solution you came up with. Good job.
That said, it looks like it can't do media right now. I would love it if it could at least give me a url for images/other media.
Feature proposal: deal with pagination.
https://krake.io/docs/define-data#next_page_object
But outside of government websites I don't see how a lot of this is even legal, per se?
Robots.txt is just a convention to advise crawlers. I'm confident most sites explicitly state this is against their terms of service.
You will encounter terms along the lines of:
"Unauthorized uses of the Site also include, without limitation, those listed below. You agree not to do any of the following, unless otherwise previously authorized by us in writing: Use any robot, spider, scraper, other automatic device, or manual process to monitor, copy, or keep a database copy of the content or any portion of the Site."
I can tell you that on several occasions I've scraped commercial sites with the permission of the owner. They want me to have access to the data but don't have the time or ability to create a proper API.
This service looks great, good work! But since you seem to host the APIs created how do you plan to get around the centralized access issues? Like on the chumby we had to do a lot of web scraping on the device itself (even though doing string processing operations needed for scraping required a lot of hoop jumping optimization to run well in ActionScript 2 on a slow ARMv5 chip with 64mb total RAM) to avoid all the requests coming from the same set of chumby-server IP addresses, because companies tend to notice lots of requests coming from the same server block really quick and will often rate limit the hell out of you, which could result in a situation where one heavy-usage scraper destroys access for every other client trying to scrape from that same source.
Having a panel for webmasters along with that would be fine.
I wrote a library for that.
https://github.com/KrakeIO/resque-my-aws
I love how you can actually use their "web scraper for anyone" on the blog post. Very cool!
>According that web site's data protection policy, we were unable to kimonify that particular page.
Sigh... Oh well... Back to scraping.
The course catalog is public, so no login is needed. I want to scrape various data related to courses, to populate forms automatically and such.