You can use portia[1]. You can try out a hosted version at scrapinghub[2] or you can host your own. It's a web based spider creator like kimonolabs but the spiders it creates are just scrapy spiders.
If you need to do something that the UI can't handle it's not a problem, you can extend it with a little python.
I agree. I actually referred people to Kimono. I'm so sorry I did. This is unacceptable behavior in this day and age. I mean seriously? Servers are cheap. Keep the thing up for existing users. Don't add new features. Don't invest any more time in the thing. But don't pull the rug from under people's feet. That's just bad form. If this is who Palantir acquires or acquihires, then they have some really bad judgement too. It reflects poorly on Palantir to not care about commitment and responsibility.
It reminds me of a PC World magazine that I had. It took me over a year and a half to get around to reading it. It had an article about the 50 best websites or something like that. By the time I got around to reading it, at least 1/3 of the sites were discontinued.
Disappointing. Was my goto service as it was a joy to use and always seemed to work. Wish they would follow the Parse lead and release something open source. Desktop doesn't really cut it for API use...
As someone who used Kimono a few times before the closure...this announcement does not surprise me at all.
The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono.
Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.
"too lazy" seems like an unfair characterization. If a service lets you do what you want faster, why not use it? I'm not familiar with Kimono, but if you can just rip it out and replace it with your own code later, why not?
For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.
While as I said, I don't know anything about Kimono, I think that statement requires qualification. If the "lazy" solution allows me to iterate faster on solving the problem, then even if I need to replace it later when my requirements are more complex but better defined, it's potentially a net win (even if you have to reimplement some of the same pieces yourself later).
It wasn't really faster. The OP actually touched on that point, but let me repeat it without ambiguity:
An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does.
What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup).
Which is even reusable.
The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc."
Heh. As if that ever did, or ever could work.
> "non-programmers can do it. No code required, etc. etc."
translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...
We, as programmers, are called in to solve the "complex use-cases." So we tend to divide the world into "things that need a novel program written for them" and "trivial problems."
You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.
I'll be the first to agree that Excel "actually works" and "solves a real problem". But if you had actually used Kimono, you'd know that it was one of those tools that was just finicky enough to be unusable by "regular people". And just weak enough to not be productive for programmers.
DiffBot appears to be different (pun intended). Even when implementing your own scraper, you still have to understand the inherent structure/organization of the webpage. DiffBot appears to do that automatically, which could be a timesaver if it works reliably without significant data fidelity issues.
I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.
Just throwing this out there in case you happen to know....I've always wanted this ability to scrape stock discussion forums. Just one example (there are probably 3 to 10 forums I'd like to scrape eventually) is:
Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3...
"access to the world’s most important data problems"
I'm not sure how to put this, but I've used Palantir's products in a couple different warzones. I can't imagine they're not pointing those tools at everyday Americans. In fact I'm certain of it (just read FedBizOpps postings from the DHS and DoD). I'm sure most of the people working there have no idea, but I'm also sure there are some truly bad people working for that company. Man, we really need to regulate this a lot better.
Powered by Palantir, basically it is real Big Brother brain network where all the information like lic. plate readers, video cam feeds, cell phone data, Internet data and everything else is "fused" together. One of the immediate resulting products of it is generation of Suspicious Activity Reports. Hoover (and Beria) would die of envy.
As a team, we’re very proud of the product we built, now used by over 125k developers, data scientists and businesses" ... from FAQ: "service will shut down Feb 29, 2016"
In fact, they are so proud, they're giving all but ~2 weeks notice to said 125K developers, data scientists, and businesses to migrate off the platform? That is an abysmally small amount of time and if I was a paying customer, i'd be furious.
Sort of plays right into the whole argument being made over on another thread (https://news.ycombinator.com/item?id=11105198) that companies shouldn't rely too heavily on proprietary tools to solve their data challenges.
Palantir is the poster child for black box software company that keeps trying to sell you more expensive licenses for their mystery algorithm. I know they've had some success in the government sector but I've seen them make a real mess of things in the private sector. At my last company their system just became a giant paracite where data went in and stuff came out the other end but it was entirely unclear what happened in between. The more we looked into the mystery middle ground the uglier it got.
There is no mystery algorithm. It's just cheap college labor working obscene hours to pump out "data analysis" scripts. The sad thing is that it actually works. The places that hire palantir are so mired in bureaucracy and process that whatever palantir builds seems to be better. Of course, just like SAP and Accenture you're now kinda locked in with support contracts.
Modern tools like spark and mesos in theory let you do what palantir does and then some but it assumes your internal IT team is semi-competent which more often than not is not a correct assumption.
Maybe it tops out early, but I'm given to understand junior salaries are also very competitive. (Maybe this is less true by comparison in the States -- it's fairly true here in the UK.)
Kind of bummed on the two weeks notice, but I'm sure there's some pressure and tradeoff the team had to make, which I guess softens the blow a bit (for me at least).
Yup. I signed up with an account to play with it once. Never used it after the initial week. I wonder how many others of that 125k were just drive by test drives.
This is really bad news for us, and the tone of the announcement grated. Two weeks notice is very bad form, I'm having to come back from vacation to fix this.
When we evaluated options a year ago, kimono was the best fit. We'd happily have paid, and did ask. I don't understand what happened... Did they run out of funding?
Here is a statement that can give you some insight
"Kimono Labs explained that in the two years since launching, it wasn’t able to have the impact that it wanted, and that the acquisition by Palantir will give the team “unmatched support, resources, and the ability to work on things we could not tackle alone as a small startup.”"
from http://venturebeat.com/2016/02/15/palantir-acquires-kimono-l...
We at CloudScrape.com are happy to welcome you onboard our platform - you get 20 hours of scraping runtime for free and the ability to navigate much more complex websites, opening up tons more opportunities then you've had previously. And we're here to stay :)
Kind regards,
Henrik Hofmeister, CTO, Co-Founder @ CloudScrape.com
We've had enterprises that depended on Kimono come to us; which isn't surprising anymore :) Hope others will give Grepsr a try as well! - https://www.grepsr.com/
If you have been burned by free services like this shutting down, give https://scrapehero.com a try - we love our customers and they love us back. We dont plan to abandon them with a 2 week notice.
ParseHub is much better at getting websites from shitty, complicated sites. Kimono was great for something simple, but I found ParseHub to be a more powerful and flexible web scraper.
Sad to see Kimono close down. I enjoyed using it for a small project, and was waiting to see a paid version with private APIs, but it was always "coming soon"...
I don't quite get why startups are not trying to generate revenue from day one.
Palantir is one of those organizations that makes me throw up a little in the back of my throat whenever I hear about them. They are sort of like the Nickelback of companies.
For morality reasons I would never be comfortable working there. The workplace could be heaven on earth and pay really well but there are simply things that are more important.
The worst part about Palantir is their ability to masquerade as this hip startup, as though working for them is as innocent as working for Imgur or Twitter or whatever. They recruit the hell out of MIT students, and it sucks seeing my friends interview for this incredibly shitty company that has openly said it won't go public since that would make running their business "very difficult" (hm, because you're doing reprehensible work perhaps?).
I wish people picked their jobs to match their ethics instead of the other way around. It makes me sad when I hear about my friends going to intern for this place.
Interning for Palantir is a pretty big resume boost for college students, and they have one of the highest paying internships in the valley. There's some serious benefits to going to Palantir.
If you have the ability to intern at Palantir you have the ability to intern at any number of more responsible companies. Palantir has a bigger booth at the career fair, but I'd bet working at Apple or Uber is just as much of a resume boost at much less of a societal cost. (source: am a college student)
Interning at Apple or Uber has much more stiff competition though. The barrier to entry for Palantir is remarkably low considering their pay, it's not surprising that college students apply for Palantir. (Source: am college senior)
Apple and Uber are not as much of a resume boost (in Silicon Valley, to a SV recruiter). Palantir is sort of on the level of quant finance / Dropbox 2 years ago.
Apple and Uber are hiring like crazy and so it's quite a lot easier to get in there as an intern than at Palantir (which is more selective).
That's surprising, as they're known for being incredibly shady and have a pretty bad rep. They might pay a lot, but it's because it's less desirable to work for the Evil Empire.
Maybe they are useful for students to signal their lack of ethics to employers? “Look, they once worked at Palantir, so they’ll have no problems doing whatever shady stuff we tell them to!”
Interning as a prison ward in Auschwitz is a pretty big resume boost for anyone who served in a prison system, and Nazis have one of the highest paying internships abroad. There's some serious benefits of becoming a prison ward in Auschwitz.
That's a rather large downgrade to the previous accusations. From evil company to a company with no benefit to mankind. I think you can say that about a lot of peoples jobs - what's your point?
My wife is a student and looking for internship she will work for a company that helps people write about how they ran over than grandma if she gets a legal. Morality and ethics have a price always.
They have one of the worst work/balances in the valley, fratty/kool aid culture, claim to be "making a difference" but really what they actually work on would be a major downside for most normal people. For example, they make targeting software and are very likely responsible for assassinations by the US and when we disappear people. They're also working on big brother tech if that's your thing.
Their YouTube channel has 100's of videos that show what they do. The one that blew my mind: GovCon7: Introduction to Palantir: https://youtu.be/f86VKjFSMJE
> When intruders from the hacker group Anonymous gained access to thousands of emails stored on the servers of the security firm HB Gary Federal, the emails revealed that Palantir had worked with HB Gary Federal to develop proposals for attacking WikiLeaks’ infrastructure, blackmailing its supporters and identifying donors.
The sex accusations ended up being baseless, and Stanford reversed their ban on Lonsdale entering the campus (and Stanford has a lower evidentiary standard than the courts)
Also, your quote doesn't appear in that article at all any more.
As for the sex accusations, who knows what really happened. Though the article doesn't exactly paint him- edit- either of them- in a flattering light. It does show poor judgement for the dude to get involved with a student he is mentoring. Then again, he also stumped on the campaign trail for Rand Paul of all people and also pals around with Mitt Romney.
> it won't go public since that would make running their business "very difficult" (hm, because you're doing reprehensible work perhaps?)
Well, more generally, anyone who has a nonstandard set of ethical beliefs, and is basing their company around that set of beliefs, can't really go public without their belief-set being thrown under the bus of their shareholders' belief-set.
For example, I've often considered starting a company in the game industry that actually hires experienced adults and treats them well, rather than hiring fresh college students and burning them out. I imagine, though, that if my hypothetical company became publicly-traded, I'd be forced to relinquish this policy in the name of short-sighted market competitiveness.
You could go the Google and Facebook route of having voting-class and non-voting class shares. Sergey and Larry, and Zuck still retain control of their companies.
On one hand, Palantir have (at best) questionable connections and I wouldn't be a rush to work for them. On the other, with the level of pay/benefits they provide, I'm not surprised they can find people without similar qualms (or people who overlook or aren't aware of them).
I won't share specifics (as they aren't mine to share), but I have a friend working for Palantir overseas. Once essential benefits (e.g. accommodation) are factored in, his salary is north of six figures (GBP). For someone only a few years past university, I imagine that sort of offer is difficult to refuse.
It's cool to get a 6 fig GBP salary but tbh, assuming what I've heard about the working hours there is true, I could still just do a combo of freelancing and contracting and make way, way more cash.
Some business owners just don't want shareholders inserting their opinion where it isn't wanted. If they don't need the funding, there's no reason to go public.
I know some engineers who work or worked there. Aside from the normal engineers (tools, backend, etc.), they are essentially high-priced tech consultants sent to companies and orgs that have no internal software talent at all, to help them analyze their (atrociously stored/managed) data. None of them work on anything shady. They essentially build web apps and backend apps to analyze and visualize data. Nothing vodoo military about it, although the military is a customer.
The pay actually isn't actually particularly high for SV standards. I think Facebook outpays them, for example - and Facebook is less selective than Palantir. But their reputation is that they're full of top engineers and you'll learn a ton there, and that's why the people I know work/worked there.
The downsides I've heard about are no work/life balance and fratty culture. Those are real downside, not the FUD about ethics.
damn. I was gonna say this means it's time to actually launch my version of this I built 3 of 4 years ago, but it looks like apifier is almost the exact same thing.....
https://webrobots.io - we give free Chrome extension to run scraper robots on your own computer. For paid customers we host robust cloud of scraping instances with all advanced features one can expect. To be fair - you need to understand javascript to write robots on our platform, we are not point-and-click doodling tool. Upside of writing robots in code - nearly unlimited possibilities with dynamic content sites, in-flight data cleaning, etc.
Two months is a short time but I had sort of expected this given how bad their service of performing last few days. That is why I had started moving to other services.
To all those badmouthing Palantir. I think they are just a hogwash. They are just pretending to be something that Fox Maulder would investigate I dont think they are. They are just helping the government spend more money on problems that are not worth solving and create new problems for citizens.
The place is likely to be very shitty to work for but by suggesting that they might be doing something that is James Bond equivalent in software world might make it sound too cool for some developers.
ParseHub is especially good at dealing with dynamic websites and multidimensional data (arbitrary relationships instead of just rows and columns).
Our team would be happy to help you migrate from Kimono. We're bootstrapped and profitable, so we don't have pressure to sell when an offer comes along. We also pledged to release the back-end code under a liberal open source license if we were ever in a position where we could no longer provide service.
Awesome. Thanks for sharing your service. My current project has been faltering because of my ignorance of these types of tools. Yours looks like it's going to make a huge difference for me!
Parsehub is easily the best hosted craping service on the market. We investigated everybody from kimono to import.io and if you want it done right they are the only option. Furthermore the guarantee of an open source version should they ever decide to close is extremely reassuring.
Its a two edged sword, sometimes you're doing great - but the fact they're shutting down means they're not doing great but have a team of experts in parsing that will benefit Palantir since they're likely into analysing surface and deep web data and just data in general :)
basically its like hiring a bunch of regex gods because you have regex problems on large scale
Disappointing to see just 2 weeks notice for their existing users.
* shameless plug * For longer than most such tools, we've been offering what our customers call the "easiest scraper of 'em all": https://feedity.com Try it out and see if it works for your needs.
Kimono had a beautiful chrome extension which allowed me to easily extract things quickly without needing to write any selectors / scraping. Much love to this team; best of luck!
> Will Palantir (or anyone else) have access to my personal data?
No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016.
Uh...but what about the scraped data? I don't think I've used Kimono since it was first announced on HN as a side project...but I'm assuming that you got to save scraped data semi-privately with Kimono? Is that considered "user data"? Kind of a vague term, and one I usually associate with data of the personal information variety. I'm sure someone at Palantir can find some interesting meta-insights from analyzing the data collected by the kind of people (data scientists, etc) who would use Kimono heavily.
140 comments
[ 1.7 ms ] story [ 1305 ms ] threadDoes anyone have any suggestions on how to run something similar on my own server?
[1] https://github.com/scrapinghub/portia [2] https://scrapinghub.com
The target demographic was developers who were too lazy to write simple scrapers for a given predefined use case. However, scraping most modern websites is not simple, and requires a lot of code to ensure the correct HTML elements are scraped and the data is transformed correctly...which defeats the purpose of using Kimono.
Kimono joining Palantir was likely an acquihire, especially since the core service was permissively free.
For example, I hit an issue when using Kimono where it refused to accept a given regular expression as valid, even after trying testing many variations which passed a regex checker. If I had just started with Python/BeautifulSoup, I could have completed the entire scraper in that time.
An actual parser for the complex use-cases requires some custom code and isn't covered by what Kimono does.
What Kimono does can trivially be achieved by writing 15 lines of python (after importing httplib2 + BeautifulSoup). Which is even reusable.
The only real use-case they had was the visual interface, so that "non-programmers can do it. No code required, etc. etc." Heh. As if that ever did, or ever could work.
translating for the Palantir's business - "a DHS/FBI/CIA analyst can dot it. No code required" Easy scraping websites for terms like "subversion", "Arab", "airplane" (cue starting scenes from Harold & Kumar 2) ...
You know what else has a visual interface? Excel. It works great! For "trivial problems." It gets stretched all the time into places where programming would have been faster, but for the "trivial problems", just being able to have a domain-expert non-programmer use Excel to implement the solution directly, is a lot faster and simpler than calling in a programmer and attempting to fully specify the problem to them.
I put the HN front page through their Discussion API Test Drive...and it identifies submission elements except submission score and plain submission URL. Which is a problem.
http://www.stockhouse.com/companies/bullboard/t.y/yellow-med...
Is this the type of task DiffBot could theoretically accomplish? (Running it through the API test drive suggests no.) Are there any other tools that might work, or is this a problem that requires hand-coding?
If you'd like me to set you up with a demo, feel free to email me: dru@diffbot.com
All of our automatic APIs aren't meant to be used on directories or listings pages (for that, we provide a crawler which gathers those links and then feeds them into our automatic APIs). The discussion API is intended to be used on discussion pages, for instance: http://www.diffbot.com/testdrive/?api=discussion&url=https%3...
If you wanted just a list of submissions and score, you could use our custom API toolkit(similar to Kimono): http://www.diffbot.com/products/custom/
I'm not sure how to put this, but I've used Palantir's products in a couple different warzones. I can't imagine they're not pointing those tools at everyday Americans. In fact I'm certain of it (just read FedBizOpps postings from the DHS and DoD). I'm sure most of the people working there have no idea, but I'm also sure there are some truly bad people working for that company. Man, we really need to regulate this a lot better.
http://www.dhs.gov/state-and-major-urban-area-fusion-centers
Powered by Palantir, basically it is real Big Brother brain network where all the information like lic. plate readers, video cam feeds, cell phone data, Internet data and everything else is "fused" together. One of the immediate resulting products of it is generation of Suspicious Activity Reports. Hoover (and Beria) would die of envy.
Could someone give an example?
Why is access to these most important problems limited to Palantir, or otherwise not accessible to most people (as implied by this shutdown notice)?
As a team, we’re very proud of the product we built, now used by over 125k developers, data scientists and businesses" ... from FAQ: "service will shut down Feb 29, 2016"
In fact, they are so proud, they're giving all but ~2 weeks notice to said 125K developers, data scientists, and businesses to migrate off the platform? That is an abysmally small amount of time and if I was a paying customer, i'd be furious.
On the other hand, congrats to Kimono team!
Why? Money.
Palantir is the poster child for black box software company that keeps trying to sell you more expensive licenses for their mystery algorithm. I know they've had some success in the government sector but I've seen them make a real mess of things in the private sector. At my last company their system just became a giant paracite where data went in and stuff came out the other end but it was entirely unclear what happened in between. The more we looked into the mystery middle ground the uglier it got.
Modern tools like spark and mesos in theory let you do what palantir does and then some but it assumes your internal IT team is semi-competent which more often than not is not a correct assumption.
My guess, they weren't commercially successful, and aquihire was the best way out.
The tone of the announcement is in bad taste given the bad news for their customers.
When we evaluated options a year ago, kimono was the best fit. We'd happily have paid, and did ask. I don't understand what happened... Did they run out of funding?
http://www.datascraping.co
We at CloudScrape.com are happy to welcome you onboard our platform - you get 20 hours of scraping runtime for free and the ability to navigate much more complex websites, opening up tons more opportunities then you've had previously. And we're here to stay :)
Kind regards, Henrik Hofmeister, CTO, Co-Founder @ CloudScrape.com
http://cloudscrape.com
I don't quite get why startups are not trying to generate revenue from day one.
edit: why the downvotes seriously.
I wish people picked their jobs to match their ethics instead of the other way around. It makes me sad when I hear about my friends going to intern for this place.
Just rail on the hypocrisy if confronted.
Apple and Uber are hiring like crazy and so it's quite a lot easier to get in there as an intern than at Palantir (which is more selective).
Interning as a prison ward in Auschwitz is a pretty big resume boost for anyone who served in a prison system, and Nazis have one of the highest paying internships abroad. There's some serious benefits of becoming a prison ward in Auschwitz.
That's how I see this parasite company.
https://en.m.wikipedia.org/wiki/Palantir_Technologies#Contro...
Also:
> When intruders from the hacker group Anonymous gained access to thousands of emails stored on the servers of the security firm HB Gary Federal, the emails revealed that Palantir had worked with HB Gary Federal to develop proposals for attacking WikiLeaks’ infrastructure, blackmailing its supporters and identifying donors.
http://www.forbes.com/sites/andygreenberg/2011/02/09/did-sec...
There was also the sex scandal with their founder and a Stanford student: http://www.nytimes.com/2015/02/15/magazine/the-stanford-unde...
This thread on HN also has some discussion about their government and military contracts:
https://news.ycombinator.com/item?id=11036156
Also, your quote doesn't appear in that article at all any more.
http://www.forbes.com/sites/andygreenberg/2013/06/07/startup...
Second to last paragraph.
As for the sex accusations, who knows what really happened. Though the article doesn't exactly paint him- edit- either of them- in a flattering light. It does show poor judgement for the dude to get involved with a student he is mentoring. Then again, he also stumped on the campaign trail for Rand Paul of all people and also pals around with Mitt Romney.
However, data mining for target leads (e.g. through cell phone, financial, social network + other data) would be much more up their alley.
Well, more generally, anyone who has a nonstandard set of ethical beliefs, and is basing their company around that set of beliefs, can't really go public without their belief-set being thrown under the bus of their shareholders' belief-set.
For example, I've often considered starting a company in the game industry that actually hires experienced adults and treats them well, rather than hiring fresh college students and burning them out. I imagine, though, that if my hypothetical company became publicly-traded, I'd be forced to relinquish this policy in the name of short-sighted market competitiveness.
Capitalism wins, eventually.
I won't share specifics (as they aren't mine to share), but I have a friend working for Palantir overseas. Once essential benefits (e.g. accommodation) are factored in, his salary is north of six figures (GBP). For someone only a few years past university, I imagine that sort of offer is difficult to refuse.
The pay actually isn't actually particularly high for SV standards. I think Facebook outpays them, for example - and Facebook is less selective than Palantir. But their reputation is that they're full of top engineers and you'll learn a ton there, and that's why the people I know work/worked there.
The downsides I've heard about are no work/life balance and fratty culture. Those are real downside, not the FUD about ethics.
http://grabby.io
http://fullcontact.com
http://emailhunter.co
http://clearbit.com
http://toofr.com
http://import.io
http://kimonolabs.com
http://apifier.com (favs)
http://elink.club
http://www.eliteproxyswitcher.com/ - ;)
https://github.com/scrapinghub/portia/ (our open source equivalent of Kimonolabs)
http://scrapy.org (allows to scrape more complex sites; also OSS)
http://scrapinghub.com (for a cloud-based platform and a smart proxy rotator)
(Disclaimer: working there.)
More seriously though, it's basically a bargain. You get:
- Access to the largest team of Scrapy, Portia, and Frontera experts around at your fingertip.
- AWS functionality for actually less than what it would cost you to set things up yourself (think VBR IP over ATM).
- Smart proxy rotating tech at your fingertip with tens of thousands of IP addresses spread across a slew of different IP networks.
- Built-in integrations with a growing list of third parties.
The list could go on and on... It's definitely not pricey. :-)
The place is likely to be very shitty to work for but by suggesting that they might be doing something that is James Bond equivalent in software world might make it sound too cool for some developers.
For folks looking for an alternative, check out http://www.parsehub.com
ParseHub is especially good at dealing with dynamic websites and multidimensional data (arbitrary relationships instead of just rows and columns).
Our team would be happy to help you migrate from Kimono. We're bootstrapped and profitable, so we don't have pressure to sell when an offer comes along. We also pledged to release the back-end code under a liberal open source license if we were ever in a position where we could no longer provide service.
https://www.quora.com/Which-are-some-of-the-best-web-data-sc...
basically its like hiring a bunch of regex gods because you have regex problems on large scale
* shameless plug * For longer than most such tools, we've been offering what our customers call the "easiest scraper of 'em all": https://feedity.com Try it out and see if it works for your needs.
There is not a download function on the desktop page, what is the direct download link? https://www.kimonolabs.com/desktop
https://www.kimonolabs.com/faq#faq-303
> Will Palantir (or anyone else) have access to my personal data?
No. All of your user data including names, email addresses, passwords and APIs will not be shared with anyone and will be securely purged from our servers on March 31, 2016.
Uh...but what about the scraped data? I don't think I've used Kimono since it was first announced on HN as a side project...but I'm assuming that you got to save scraped data semi-privately with Kimono? Is that considered "user data"? Kind of a vague term, and one I usually associate with data of the personal information variety. I'm sure someone at Palantir can find some interesting meta-insights from analyzing the data collected by the kind of people (data scientists, etc) who would use Kimono heavily.