Meetup.com has calendar feeds. You can get a calendar feed for all the events you've RSVPed "yes" to. You can get individual calendar feeds for each group's events. But there's no way to get a calendar feed of all the events happening in all of your groups.
I'm a member of something like 40 groups. The thought of manually adding each feed as a separate calendar in Thunderbird - and the thought of Thunderbird then trying to cope with the result - was... unpleasant.
So I set up a pipe to combine all the feeds into one. Very cursory googling suggested that Yahoo Pipes was the easiest way to accomplish this.
Fairly slick UI. Unfortunately it turns out that meetup.com apparently serves 503s to requests coming from Pipes.
Unfortunately it turns out that meetup.com apparently serves 503s to requests coming from Pipes.
That's really interesting. I think it exemplifies a problem that has been bothering me for a while: for many sites, scripting and design are antipatterns designed to corral rather than enable the user and leverage branding.
I used a pipe that created RSS feeds from Google+ pages in conjunction with some other stuff to get all the articles I'd want to read in one document (with a table of contents) delivered to my kindle.
I'm now working on something that doesn't use Y! Pipes and simplifies the process.
We all knew it was coming, but the end of Pipes still sucks.
Oh man, I use 3 or 4 pipes to filter various RSS feeds. Ugh. Good suggestions for replacements (especially those that don't require my own server) would be much appreciated.
My understanding is that YQL is being used as the middleware between services internally. Will probably not be shut down, but they could shut off the publicly accessible one.
One thing not mentioned in this thread is that Pipes 2.0 was a rewrite from the original Pipes which was written in Perl. We ported it to the same engine that powers YQL.
As for why shut down Pipes if it shares infrastructure with YQL? Probably because usage is mostly RSS crawlers that no one looks at anymore.
YQL queries can do everything pipes can, you just have to write little scripts to do it instead of the UI
Argh! The UI was the innovative part. The whole point of something like this is that you don't have to learn a bunch of syntax - when you add a module, it tells you what sort of data it needs to know (such as a URL or...) and when you connect modules together it tells you what sort of inputs and outputs are available. This allows you to focus on the actual problem domain instead of the morphology of the query language, which is not important to the user.
I keep encountering this attitude among programmers - 'I type code all day, so why don't you want to do the same thing?' If people wanted to type stuff all the time the command line would still rule, but the majority of people prefer graphic user interfaces because they don't want to write code, they want to make selections from available options and use the computer to take care of the plumbing. This is a Good Thing. It's easier, more fun, and yields greater productivity, for the same reason that handing someone a box of Lego is better than handing them a few blocks of wood and a set of carving tools. The existence of Lego hasn't led to a collapse of sculpture or mechanical engineering. On the contrary, the simplicity, consistency, and interoperability of the different components has made a massive creative, educational, and commercial success.
I don't mean this as a hit on you personally, but I find comments like the above frustrating. Imagine you went into a police station to report the disappearance of your bicycle and the person behind the desk shrugged and said 'you're still able to run around on your legs, what's the problem? Your bicycle may have allowed you to go faster down the street but it's not like you could ride it up and down stairs so you should be happy with walking on your feet.' The problem is the loss of a tool that offered substantial efficiencies on some common tasks even though it wasn't ideal for every situation. You'd probably be equally pissed off if your favorite high level language went away and you were told you could just go back to writing assembler. I personally love doing things in assembler because it makes me feel smart but that doesn't mean it's better than high level languages.
Yeah, I wasn't suggesting that YQL was equivalent to Pipes, just that it might be able to replace the functionality. Which is why I made sure to mention the significant difference...
I get the impression that YQL is suffering from the same kind of neglect Pipes did. It was down multiple times a week for me, and I had to move to other solutions.
pipes creator here: sad to see pipes shutdown. i am surprised it lasted this long -- it was abandoned years ago. on the positive side, it is nice to see it mentioned in the same sentence as y! maps which used to be a big deal back in the day.
It really would be oh so nice to have something that can ingest RSS and Tweets and then apply filters to them while visualizing the logic through a graphical interface. Maybe even some fuzzy machine learning to use in building filters. Some more advanced dupe technology would go a long way. People would be building their own techmemes in no time. You could build ranking systems that award bonus points if something like longform or techmeme mention something.
Maybe im dreaming about a free version of Percolate, which I have heard is really cool.
One of our open source projects[1], lets you consume RSS, Email, and Tweets, with more "stuff" coming, and apply filters and set triggers, etc. It's still pretty raw, and we don't have the really awesome visual interface for laying things out, but that sort of thing is something we hope to get to eventually. One other neat thing we do is pipe content through Apache Stanbol to do semantic entity extraction, and build up a knowledgebase as content is consumed.
This is exactly what we are doing at http://www.feedsapi.com , we don't use a graphical UI for the logic but keywords and RegEx to create filters... Many folks are already building their own techmeme with our API, it's however not a free service. I'm not sure if this fully fits your requirements but you should take a look.
Founder of Enginuity Analytics here. You can ingest RSS and Tweets into our platform where we apply machine learning to provide social data, sentiment analysis, rankings, filters and demographic analysis of all the content for 27 languages. And easily export the data to Excel and more. http://theenginuity.com
Instead of closing it down, releasing the source and giving it to the community could breath live into it.
I remember being blown away by it, the fact that it was neglected and just kept on says a lot about it's good design.
This is an amazing product. Have you tried talking to VCs or anyone about someonr adopting it somehow?
I wonder if it would be possible to build something similar using IPV6/some kind of P2P/STUN/TURN/UDP hole punching/swarm/ethereum / any way to make a distributed system that does the same thing. Or maybe just volunteer running Node on their own VPSs.
Several reasons I can think of. For one, no one at Yahoo seemed to understand what Pipes was for (part of a wider problem where Yahoo didn't seem to know what the company was for).
For another, Yahoo legal prevented us from doing any kind of long-term storage of RSS feeds. That made the service far less useful than it could have been.
Finally, they didn't give it any resources at all. Management was too preoccupied with the failed Brickhouse project (internal startup incubator) and the host of failed ideas that it spawned.
Filtering and triggering without the annoyance of having to write code in some syntax-of-the-month or dredge through poorly written documentation for half-ass APIs.
If you are redistributing, hosting or using it commercially you have to have the agreement of the author of the content. RSS just gives you access in another way than HTML, but it does not circumvent copyright.
Let me just tell you and anyone that worked on the team, thanks so much for it. You saved me hours of work and let me do it really damn well. This is a sad, sad day. I used Pipes extensively and ironically enough at Yahoo for some features/prototypes on News and Search. It was such an elegant and powerful app, I really wish Yahoo had given it more love.
Pipes has long been a sort of monument to the faith of a bygone age -- the first flush of Web 2.0 optimism, when standard formats and open APIs were going to let us mix and match and mash up services of all kinds at will.
That faith is dead now, of course, scoured from the earth by walled gardens and VC money. But like Catholics in Elizabethan England, some of us quietly tend our secret shrines and pray for its return.
(Hopefully that will work out better for us than it did for them.)
Thank you for your comment, for a (long) while I thought that I was the only one remembering those times and especially the ideals people were bringing up then. Sometime in 2005 I even copy-pasted this quote on my blog, high in believing that the future will belong to open data and open information:
> Bosworth advocated an open model for data.(...)
Imagine if you can query any data that is available anywhere in the world. Bosworth said that what this requires is a single, simple, open wire format for items. The format needs to be simple for any P programmer to deliver and any JavaScript programmer to consume. He also pointed out that "complex things tend to break and simple things tend to work." Google has the simplest query language in the world. There is no structure and no syntax. (my note: at that time Bosworth was VP of Product Management at Google)
Love your analogy, sure made me laugh (visualising a geeky dev like myself tending a weathered gnarly monument in a secluded English glade... in real life, that is, not in an RPG).
However, why such pessimism? I hardly think that standard formats and open APIs are dead. Sure, there are some nasty blights upon cyberspace in this regard (ahem, Facebook, Apple), but other players are still keeping the dream alive to some extent (e.g. Google, Yahoo).
I'd count google mostly as an example of deprecating or limiting useful interfaces. Yes, you can feed them stuff in "open" formats, but getting stuff out is harder. First example that comes to mind is youtube APIs, which offered easier access to stuff. Now try to get a feed of someones videos or similar...
But it is possible that it's confirmation bias + uncommon sample on my part.
Yeah... for Google / YouTube, I blame OAuth for most of the difficulty in accessing data (while retaining one's sanity) these days. Once you've waded through the quagmire of access tokens, auth libraries (in different languages), account permissions, etc... making the actual API calls isn't so hard.
We (Webflow) are actually considering building a Pipes alternative in the near future, and I'd love to sit down with you and brainstorm some ideas. If you'd be willing to make some time to see a very rough demo, I'd love to get in touch - my email is in my profile.
There was IFTTT but looking at their page now it seems like maybe they have pivoted, at least I can't tell that it works like it used to: https://ifttt.com/
If you sign in all the previous functionality is still there. Seems their landing page is only advertising their new "Do" apps (which are a one-button extension of the old IFTTT recipes). Installed it, but never used them much.
Zapier is the only reliable tool I found so far that can still import RSS to facebook pages, it tried it with pipes and it had issues with non-standard fields.
It's amazing how much a shift of management focus can change things... Google embraces RSS, develops the premiere online RSS reader, so good that even stand alone readers are largely abandoned... then google nukes RSS support from Chrome, and shutters Reader. And the world is worse off for it.
It definitely inspires a "never forget" kind of mindset, such that I don't trust relying on a SaaS that I don't have a personal exit strategy for.
I liked the concept and probably didn't give it a fair shake, but it flooded me with notifications and I just didn't want to take the time to tweak it to my needs. Its one of those apps you need a good tutorial (from someone other than the creators) to walk you through useful functionality and setup.
The only thing Pipes would need is a .io domain and it could be dressed up as the Next Cool Thing which LargeCorp is desperate to get its hands on to prove it's still cool.
> ChangeTracker watches the White House’s web site so you don’t have to. Whenever a page on whitehouse.gov changes, we’ll let you know — via e-mail, Twitter, or RSS. But ChangeTracker is not a piece of software. It’s the output of a series of powerful and mostly free Web-based tools, lovingly connected over the Internet. Here’s how to do it yourself so you can track changes on any Web site on the Internets.
Who is working on a pipes replacement and where can I sign-up? Maybe this will be like when google killed Reader and RSS replacements became a hot area...
IFTTT doesn't let me basically reformat an RSS feed and then see it in my RSS reader. Most/all of my use of pipes was "go scrape this thing and turn it into RSS, filtering out stuff I don't want to see."
It was great for that - I'm unaware of anything that is even trying to replace it.
I was looking into doing a Pipes 2.0 sort of project. If you live in the bay area i'd love to get coffee and chat about it. There are a number of the pieces to pipes that are opensource, but none of them are strung together.
I also thought that pipes queries were amazing, but slow, so it was hard to use them as part of an application.
We're working on something that covers a lot of the same ground, although it's not exactly the same thing. It's also not very polished right now, but we definitely hope to get to that level eventually. It's all open source, Apache Licensed, and built using Grails. We don't have YQL, but we extract entities from the content and store it in a triplestore that can be queried using SPARQL, and eventually the SPARQL support will be woven more completely through things.
Shameless relevant plug: http://www.feedsapi.com might be an option, it'sit's not exactly a replacementbfor pipes but it does a lot of what pipes did and is a work in progress
This a disappointment if not an unexpected one. I used pipes again just a few weeks ago to do some simple modification of feed but thankfully ended up taking another approach (not a pipes replacement I just wrote some code for it). Pipes was a really cool tool and it will be missed. Does anyone know of/use any alternatives?
I've used this pipe as my Hacker News RSS feed for the last 5 years, which flips the link structure so the main link for each entry is to the comments.
The main one I made was to switch the actual link for the Reddit RSS feed to be the main link, set the true article title as the item title, use the submitted title as the item body and add a link to the link's domain in the bottom.
I still have 146 subscribers to keep happy, though Feedburner should make any transition transparent. I'm disappointed it's going. Thank you to those who made it!
I worked on Pipes briefly, shortly after it launched. It was really two distinct products rolled into one. The visual interface was groundbreaking and stunning-I think half my team was hired on the strength of that demo. No one could believe Javascript could do that.
The backend was an extremely useful tool for munging RSS feeds. With any kind of support, or even benign neglect, the product would have been successful. It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.
Hats off to psadri and the other Pipes creators for a really stellar piece of work.
Ahh, what a terrible CEO. When the only two possibilities to explain your tenure are 1) a lying co-conspirator or 2) mind-blowingly incompetent--then you've truly done a bang-up job.
By the way: many people in the past, including here on HN says that Elop was right, and that the reason Nokia died was 100% apple+google fault.
Here in Brazil Elop effect was interesting: it was VERY obvious, yet noone here knows who the hell Elop is.
To put it simple: When Elop released that stupid memo, Nokia had 67% of Brazil smartphone market share, and the share was still rising, and Nokia utterly dominated in the non-smartphone market too. Right after Elop memo, the first thing that happened is that suddenly the communities died (although Nokia store DID suck, there was at least in Brazil a vibrant freeware app community that shared stuff in forums), then just some months later you could not find phones for sale, even new models, because the import companies DID heard of Elop memo, and decided to not risk importing the phones, even if shopkeepers still wanted it.
The end result was funny: Elop nuking of Symbian was so hard, that in Brazil other smartphones could not keep up for a while, and we temporarily had a decrease in smartphone use, and a increase in feature phone use, and that increase came mostly from truly crap chinese companies, because not even Nokia feature phones could be found in stores anymore.
> Elop was a director of consulting for Lotus Development Corporation before becoming CIO for Boston Chicken in 1992,[3][7] which filed for Chapter 11 bankruptcy in 1998
> In the same year, he joined Macromedia's Web/IT department[7] [...] from January 2005[10] for three months before their acquisition by Adobe Systems
> He was then president of worldwide field operations at Adobe, tendering his resignation in June 2006 and leaving in December,[12] after which he was the COO of Juniper Networks for exactly one year from January 2007 – 2008
This is not the CV of a man that I would hire for anything other than running a going concern into the ground.
i think nokia is more about lack of executive vision/market knowledge. The other examples are more about middle management, besides the executive lack of vision and bean counting.
Is Nokia's example really good ? Nokia remain one of the largest companies in telecom hardware provider and handsets was always a side business for them. They successfully sold it to microsoft.
I'm still upset by the fact that MeeGo/Maemo was killed, and Nokia sold to Microsoft. The N9 was still the best phone I've ever used, and it's UX ideas are still the simplest and most effective way of using a phone, and yet no one else has implemented it.
Jolla (https://jolla.com) was founded by people involved in MeeGo development. While their phone got a lukewarm response, I've heard many good things about their upcoming tablet and have ordered one. As you used MeeGo, how does Sailfish OS appear to compare with it?
Sailfish unfortunately does not have all of the UI things I'm talking about -- the "three screen with edge gestures", which is a shame :( They couldn't, because the N9's UI was never open sourced despite the core being so.
Incorrect. The first rule of mismanagement club is to never know exactly how you got into mismanagement club; but by god you are going to do a bang up job of it now you are in!
I think most/many people could reasonably say this about their company; I currently work at Google and feel quite the same. I'm not convinced that either Google or Twitter[1] qualify as a particularly spectacular example of mismanagement though.
[1] I don't have first-hand experience but one of my best friends work there and we regularly vent to each other about work complaints.
I wrote a simple Wordpress plugin that works with RSS feeds. In addition to aggregating fees, Pipes has been a go-to solution when people report feeds that are malformed, have some weird characters or anything that chokes the Wordpress RSS parser (simplepie). People would run them through pipes, which seemed to handle anything and output clean RSS.
I'm sad to see it go and it'll be missed by quite a few people who use my plugin.
I'll echo those sentiments, the UI for that time certainly had a wow factor.
> With any kind of support, or even benign neglect, the product would have been successful.
Look at IFTTT services now and it's easy to realize that pipes even in its original incarnation could have been so much more given its brain dead UI ease.
It's a shame that Yahoo is shutting this down rather than pivoting it as an IFTTT or similar. I'd wager that it's lack of ownership that makes it hard to pivot. I also don't recall it having a premium tier which has got to weigh down the resources--talent always gravitates toward revenue centers it seems.
> talent always gravitates toward revenue centers it seems.
Ie the 'new management can't fire my entire division for being unprofitable' center. Seems like a smart move for the people who wrote Pipes but never found a business model for it.
I can understand why Pipes would be shut down by Yahoo, it was a impressive technical feat but it definitely appealed to very small developer focused crowd.
I feel like IFTTT is our best bet, while it doesn't do the "Unix pipe" that pipes offered it at least provides some usability to the process.
Sad but not expected. It was really powerful but, like most people, I stopped using it years ago. It kept breaking either due to the sites I was using it with changing things or the service being neglected (and my limited ability to understand how it worked!)
If I was looking for a replacement, maybe I'd look for an RSS aggregation plugin for WordPress? Personally, I have my own PHP code that will slice and dice feeds however I want.
I got about halfway through building a "news blend" app where you tap just the categories you want and it would blend all the latest articles from just your categories into one news feed and email that to you daily. But we never finished it.
Pipes was the first service I ever owned from an ops perspective (circa 2008). We had two clusters, one on each coast. I thought we had capacity to take one offline to do a full upgrade, turns out we didn't.
By the time I turned the west coast back online, the east coast was failing health checks, and the LB failed everything over to the west coast, which then proceeded to be overloaded, and everything flipped back to the east coast... Ping ponged for about 2 hours until they finally settled.
Fun times.
Another fun story: For the longest time the front page had an example pipe that merged search results from various online sites (amazon/ebay/cl). It was made by a former employee and was easily one of the most popular pipes. One day we found out he had his affiliate id in all of those links. We chuckled and moved on.
Haha, I loved that search results pipe! The product was amazing at the time. It's really too bad that Yahoo didn't try to make something more out of it.
Aw man - that is sad. We at Zapier took great inspiration from Pipes, it was such a cool and powerful product! It was a shame to see it languish.
I'd sure love to pick the brain of anyone formerly involved in the project - it seems like there is a lot we could learn from the trail it blazed. My email in profile, obligatory beers/coffee/etc. offer.
You guys are fantastic. I use Zapier for several of my organizations and companies, and I cannot understate how incredible of a service you guys have created. <3 for days.
My only wish would be for a paid tier between 100 and 3000. If I went hog-wild and did everything I want to do, I'd probably use 200-300 a month. I would love to pay for that. But there's some loss-aversion psychology happening that makes me not want to pay for 3000 if I know I'll only use 300.
I'd give you my cc number right now if you did a 1000 zaps at $10/m.
That being said - you know your business better than I do. If it would hurt conversions of $20/m more than it would help, I get that. :)
I actually had the same query last year when I pushed a place I work at to use a combination of Yahoo! Pipes and Zapier for our Facebook postings. I asked for a smaller plan (we only need about 600-700 zaps/month at the most) and was just instructed by their support to utilize their reward program[0] which offered an extra 50-100 zaps/month based on what you 'complete,' and it included things like (IIRC) liking them on Facebook or following them on Twitter.
It seems like that program has since been limited, however, as it now redirects to a referral page[1] only. A combination of a couple referrals around the office and performing the required tasks gave us about ~650 monthly "zaps" at the time and we haven't needed more than that so far. Still, I would have happily paid $5/month or something to avoid the hassle (and be able to pay them for their wonderful product).
Aw damn, that kinda sucks, but the project has been dead for a while now. I used it for content aggregation, but moved on to custom scripts, which were way more powerful...
Most of what Pipes did can be done locally, in a library. FeedReader for Windows did this.[1] Unfortunately, they recently announced "Get ready for the brand new FeedReader! ... Check our site later this March for a new amazing web service, featuring industry-best RSS search and analysis.", which means it's becoming "cloud based" and probably has ads or worse. "rss.com" already went that way; they charge $6/month to read RSS feeds.[2]
Following RSS feeds on a continuing basis takes a lot of RSS polls. Most RSS feeds do not implement RSS in a way that allows getting only new items reliably.[3] The RSS "etag" mechanism is not reliable. Some sites with multiple servers and a load balancer have different etag values on each server. The "guid" field sometimes changes when the content hasn't changed. My experience is that nothing short of full text comparison eliminates duplicates properly. I wrote an RSS reader which does a MD5 of the text of each incoming message to throw out duplicates. Presumably the "pipes" system did something similar.
If RSS feed servers complied with the standard, there'd be less need for feed aggregation services.
I think your first paragraph speaks to a real tension in the world of software: web applications have enormous discoverability, immediacy, upgradeability, and (often) social collaboration advantages over native applications, but they have ongoing costs proportional to their popularity, so it's harder to run them without ulterior financial motives.
Just a shout out to the awesome Pipes community and some people that really helped it along the way. All the original creators (pasha, JT, ed, daniel), community members and team: hapdaniel, dawnfoster, kentbrewster, mirek, spullara, janluehe, dspark, kevink, brettp, lolo, laurencecoates, earth2marsh, psychemedia, davglass, ssaine, sadaf, ido, nagesh, ameya. I know i'm missing tons more, but just top of my head. You are all awesome.
One pipe would grab data from Craigslist, strip out irrelevant items, and send me a text if anything new and interesting appeared. Ditto for Westside Rentals.
This combination worked great. If a lame apartment appeared in CL, I'd edit my Pipes regex to strip the same thing out of WR. In this way I'd get only texts for awesome places.
Ah, I did this too! I'd forgotten about it. As I was looking, I'd continually update it to block out streets I wasn't interested in, then eventually I was only getting notifications for exactly where I wanted for exactly how much.
I also used it with Freecycle and got a fridge-freezer! (you had to be quick to get them)
I was also using it exactly like this until recently! I sent my roommate the pipe and we'd each watch/browse it for new places. It saved us a ton of effort.
183 comments
[ 3.3 ms ] story [ 218 ms ] threadI used it for the first time 2 weeks ago.
Meetup.com has calendar feeds. You can get a calendar feed for all the events you've RSVPed "yes" to. You can get individual calendar feeds for each group's events. But there's no way to get a calendar feed of all the events happening in all of your groups.
I'm a member of something like 40 groups. The thought of manually adding each feed as a separate calendar in Thunderbird - and the thought of Thunderbird then trying to cope with the result - was... unpleasant.
So I set up a pipe to combine all the feeds into one. Very cursory googling suggested that Yahoo Pipes was the easiest way to accomplish this.
Fairly slick UI. Unfortunately it turns out that meetup.com apparently serves 503s to requests coming from Pipes.
That's really interesting. I think it exemplifies a problem that has been bothering me for a while: for many sites, scripting and design are antipatterns designed to corral rather than enable the user and leverage branding.
We all knew it was coming, but the end of Pipes still sucks.
Still has the Yahoo problem, but it might be the easiest short term.
One thing not mentioned in this thread is that Pipes 2.0 was a rewrite from the original Pipes which was written in Perl. We ported it to the same engine that powers YQL.
As for why shut down Pipes if it shares infrastructure with YQL? Probably because usage is mostly RSS crawlers that no one looks at anymore.
Argh! The UI was the innovative part. The whole point of something like this is that you don't have to learn a bunch of syntax - when you add a module, it tells you what sort of data it needs to know (such as a URL or...) and when you connect modules together it tells you what sort of inputs and outputs are available. This allows you to focus on the actual problem domain instead of the morphology of the query language, which is not important to the user.
I keep encountering this attitude among programmers - 'I type code all day, so why don't you want to do the same thing?' If people wanted to type stuff all the time the command line would still rule, but the majority of people prefer graphic user interfaces because they don't want to write code, they want to make selections from available options and use the computer to take care of the plumbing. This is a Good Thing. It's easier, more fun, and yields greater productivity, for the same reason that handing someone a box of Lego is better than handing them a few blocks of wood and a set of carving tools. The existence of Lego hasn't led to a collapse of sculpture or mechanical engineering. On the contrary, the simplicity, consistency, and interoperability of the different components has made a massive creative, educational, and commercial success.
I don't mean this as a hit on you personally, but I find comments like the above frustrating. Imagine you went into a police station to report the disappearance of your bicycle and the person behind the desk shrugged and said 'you're still able to run around on your legs, what's the problem? Your bicycle may have allowed you to go faster down the street but it's not like you could ride it up and down stairs so you should be happy with walking on your feet.' The problem is the loss of a tool that offered substantial efficiencies on some common tasks even though it wasn't ideal for every situation. You'd probably be equally pissed off if your favorite high level language went away and you were told you could just go back to writing assembler. I personally love doing things in assembler because it makes me feel smart but that doesn't mean it's better than high level languages.
Maybe start a new, open source version of Pipes?
Maybe im dreaming about a free version of Percolate, which I have heard is really cool.
[1]: https://github.com/fogbeam/Neddick/
Pipes would often get blocked by 3rd party data providers, which motivated me to roll my own library.
No graphical interface. Personally, I found Y! Pipes UI hard to work with when I needed to build large pipelines.
RssPercolator is a .NET library used for downloading, aggregating, and filtering RSS feeds. Developed out of frustration with Yahoo Pipes.
- RSS and Atom feed formats - Asynchronous multi-feed download - Multiple sources and multiple destinations - Feed filters (string match, wildcards, and Regex)
I wonder if it would be possible to build something similar using IPV6/some kind of P2P/STUN/TURN/UDP hole punching/swarm/ethereum / any way to make a distributed system that does the same thing. Or maybe just volunteer running Node on their own VPSs.
For another, Yahoo legal prevented us from doing any kind of long-term storage of RSS feeds. That made the service far less useful than it could have been.
Finally, they didn't give it any resources at all. Management was too preoccupied with the failed Brickhouse project (internal startup incubator) and the host of failed ideas that it spawned.
What was Pipes for?
Wait, is there something legally wrong about storing RSS feed data?
That faith is dead now, of course, scoured from the earth by walled gardens and VC money. But like Catholics in Elizabethan England, some of us quietly tend our secret shrines and pray for its return.
(Hopefully that will work out better for us than it did for them.)
> Bosworth advocated an open model for data.(...) Imagine if you can query any data that is available anywhere in the world. Bosworth said that what this requires is a single, simple, open wire format for items. The format needs to be simple for any P programmer to deliver and any JavaScript programmer to consume. He also pointed out that "complex things tend to break and simple things tend to work." Google has the simplest query language in the world. There is no structure and no syntax. (my note: at that time Bosworth was VP of Product Management at Google)
That is all dead now.
However, why such pessimism? I hardly think that standard formats and open APIs are dead. Sure, there are some nasty blights upon cyberspace in this regard (ahem, Facebook, Apple), but other players are still keeping the dream alive to some extent (e.g. Google, Yahoo).
But it is possible that it's confirmation bias + uncommon sample on my part.
What do you mean? Channels have an element linking to an RSS feed. In fact, just pasting a channel URL to an RSS reader should work.
Example of a channel feed: https://www.youtube.com/feeds/videos.xml?channel_id=UCr4soU_...
We (Webflow) are actually considering building a Pipes alternative in the near future, and I'd love to sit down with you and brainstorm some ideas. If you'd be willing to make some time to see a very rough demo, I'd love to get in touch - my email is in my profile.
Seriously, I think the only Yahoo products I use are Pipes, Groups and Flickr. And Flickr doesn't really count.
It definitely inspires a "never forget" kind of mindset, such that I don't trust relying on a SaaS that I don't have a personal exit strategy for.
https://ifttt.com/products/do/button
Their site aesthetic is trying to be Nest-y, but the UX does not communicate simplicity at all.
The only thing Pipes would need is a .io domain and it could be dressed up as the Next Cool Thing which LargeCorp is desperate to get its hands on to prove it's still cool.
> ChangeTracker watches the White House’s web site so you don’t have to. Whenever a page on whitehouse.gov changes, we’ll let you know — via e-mail, Twitter, or RSS. But ChangeTracker is not a piece of software. It’s the output of a series of powerful and mostly free Web-based tools, lovingly connected over the Internet. Here’s how to do it yourself so you can track changes on any Web site on the Internets.
It was great for that - I'm unaware of anything that is even trying to replace it.
What would be the best toolset to recreate the Pipes UI?
It's pretty slick, and I could see the underlying UI being useful for modeling a variety of things...
I also thought that pipes queries were amazing, but slow, so it was hard to use them as part of an application.
https://github.com/fogbeam/Neddick
http://pipes.yahoo.com/pipes/pipe.info?_id=0e19519adf0515251...
Thanks to whoever made it. I guess I'll have to find a way to do it myself now.
http://feeds.feedburner.com/RedditArticles
And another where I put the pictures inline (though the 3rd party API I was using isn't handling things too well anymore)
http://feeds.feedburner.com/RedditPicturesWithPictures
I still have 146 subscribers to keep happy, though Feedburner should make any transition transparent. I'm disappointed it's going. Thank you to those who made it!
The backend was an extremely useful tool for munging RSS feeds. With any kind of support, or even benign neglect, the product would have been successful. It took a lot of active mismanagement and folly to keep Pipes from living up to its promise.
Hats off to psadri and the other Pipes creators for a really stellar piece of work.
Pipes's interface is primary the work of Jonathan Trevor, with lots of help from Daniel Raffel, Ed Ho and others.
I feel like this unique blend will be Yahoo's lasting contribution to the field of management theory.
Here in Brazil Elop effect was interesting: it was VERY obvious, yet noone here knows who the hell Elop is.
To put it simple: When Elop released that stupid memo, Nokia had 67% of Brazil smartphone market share, and the share was still rising, and Nokia utterly dominated in the non-smartphone market too. Right after Elop memo, the first thing that happened is that suddenly the communities died (although Nokia store DID suck, there was at least in Brazil a vibrant freeware app community that shared stuff in forums), then just some months later you could not find phones for sale, even new models, because the import companies DID heard of Elop memo, and decided to not risk importing the phones, even if shopkeepers still wanted it.
The end result was funny: Elop nuking of Symbian was so hard, that in Brazil other smartphones could not keep up for a while, and we temporarily had a decrease in smartphone use, and a increase in feature phone use, and that increase came mostly from truly crap chinese companies, because not even Nokia feature phones could be found in stores anymore.
By the time Elop was announced as CEO in late 2010, Kallasvuo had already guaranteed Nokia was toast.
Just go back to the way Nokia was behaving, and what they were saying in 2007/2008 for easy proof of that.
"Nokia CEO calls Apple iPhone ‘niche product’"
http://macdailynews.com/2008/04/17/nokia_shares_slammed_in_w...
http://www.informationweek.com/mobile/nokia-ceo-disses-iphon...?
http://www.forbes.com/2007/02/12/nokia-kallasvuo-iphone-face...
http://www.forbes.com/forbes/2007/1112/048.html
> In the same year, he joined Macromedia's Web/IT department[7] [...] from January 2005[10] for three months before their acquisition by Adobe Systems
> He was then president of worldwide field operations at Adobe, tendering his resignation in June 2006 and leaving in December,[12] after which he was the COO of Juniper Networks for exactly one year from January 2007 – 2008
This is not the CV of a man that I would hire for anything other than running a going concern into the ground.
I would love to read a blog post about how exactly it was mismanaged. It would be great for others to learn lessons from the mistakes.
[1] I don't have first-hand experience but one of my best friends work there and we regularly vent to each other about work complaints.
I'm sad to see it go and it'll be missed by quite a few people who use my plugin.
Jonathan Trevor's Javascript was out of this world amazing for the time.
> With any kind of support, or even benign neglect, the product would have been successful.
Look at IFTTT services now and it's easy to realize that pipes even in its original incarnation could have been so much more given its brain dead UI ease.
It's a shame that Yahoo is shutting this down rather than pivoting it as an IFTTT or similar. I'd wager that it's lack of ownership that makes it hard to pivot. I also don't recall it having a premium tier which has got to weigh down the resources--talent always gravitates toward revenue centers it seems.
Ie the 'new management can't fire my entire division for being unprofitable' center. Seems like a smart move for the people who wrote Pipes but never found a business model for it.
I feel like IFTTT is our best bet, while it doesn't do the "Unix pipe" that pipes offered it at least provides some usability to the process.
I got about halfway through building a "news blend" app where you tap just the categories you want and it would blend all the latest articles from just your categories into one news feed and email that to you daily. But we never finished it.
By the time I turned the west coast back online, the east coast was failing health checks, and the LB failed everything over to the west coast, which then proceeded to be overloaded, and everything flipped back to the east coast... Ping ponged for about 2 hours until they finally settled.
Fun times.
Another fun story: For the longest time the front page had an example pipe that merged search results from various online sites (amazon/ebay/cl). It was made by a former employee and was easily one of the most popular pipes. One day we found out he had his affiliate id in all of those links. We chuckled and moved on.
Just to be clear, I had not idea about that until now.
Wow that must have added up to quite the haul
I'd sure love to pick the brain of anyone formerly involved in the project - it seems like there is a lot we could learn from the trail it blazed. My email in profile, obligatory beers/coffee/etc. offer.
My only wish would be for a paid tier between 100 and 3000. If I went hog-wild and did everything I want to do, I'd probably use 200-300 a month. I would love to pay for that. But there's some loss-aversion psychology happening that makes me not want to pay for 3000 if I know I'll only use 300.
I'd give you my cc number right now if you did a 1000 zaps at $10/m.
That being said - you know your business better than I do. If it would hurt conversions of $20/m more than it would help, I get that. :)
<3 you guys though.
It seems like that program has since been limited, however, as it now redirects to a referral page[1] only. A combination of a couple referrals around the office and performing the required tasks gave us about ~650 monthly "zaps" at the time and we haven't needed more than that so far. Still, I would have happily paid $5/month or something to avoid the hassle (and be able to pay them for their wonderful product).
[0] https://zapier.com/rewards/
[1] https://zapier.com/referral/
Following RSS feeds on a continuing basis takes a lot of RSS polls. Most RSS feeds do not implement RSS in a way that allows getting only new items reliably.[3] The RSS "etag" mechanism is not reliable. Some sites with multiple servers and a load balancer have different etag values on each server. The "guid" field sometimes changes when the content hasn't changed. My experience is that nothing short of full text comparison eliminates duplicates properly. I wrote an RSS reader which does a MD5 of the text of each incoming message to throw out duplicates. Presumably the "pipes" system did something similar.
If RSS feed servers complied with the standard, there'd be less need for feed aggregation services.
[1] http://feedreader.com/ [2] https://www.rss.com/ [3] http://www.詹姆斯.com/blog/2006/08/rss-dup-detection
One pipe would grab data from Craigslist, strip out irrelevant items, and send me a text if anything new and interesting appeared. Ditto for Westside Rentals.
This combination worked great. If a lame apartment appeared in CL, I'd edit my Pipes regex to strip the same thing out of WR. In this way I'd get only texts for awesome places.
Vaya con dios, Yahoo Pipes!
I also used it with Freecycle and got a fridge-freezer! (you had to be quick to get them)