Thanks! I'm one of the devs on the project; you can contact me directly at Zane@Streamified.com.
To your questions: one of the major benefits of the service is the "crowdsourced" aspect of it. That is: the more people using the same API, the more news articles get found and indexed. Furthermore, the API actually hooks into other paid APIs, so it actually wouldn't work very well for you to run it on your own servers unless you wanted to also pay for them. For all these reasons and more, we think that a hosted API is the best solution.
The idea is good. But clarifying it further, how many API calls per month do you offer or plan ?
Correct me if I am wrong, searchable means topic/keyword parsing with context analysis (i.e. intention analysis)?
Our courtesy limit is 1k requests per day to the API. Beyond that, we're happy to discuss what makes sense, we obviously just need to prevent abuse.
Yes, our searching capacities include everything from categories to sentiment to tags to media... you name it. ElasticSearch powers our backend, which is powered by Lucene, so anything in an article can be searched. Check out the articles/search endpoint: https://streamified.me/developer/#endpoint=GET%20articles%2F...
If this depends on the individual APIs (especially twitter) and sees any sort of traction, isn't there a massive risk that one of the providers could block access to streamified?
Per our "whitelabeling" philosophy, Streamified actually uses your own app IDs (not ours). From Twitter's perspective, we're just acting as a middleman to help you use their API better; any blockage, then, would be a result of your implementation of their API (throttling, etc).
Thanks! Be sure to drop us a line if you want to dig a bit deeper into integration.
For social networks, we're proxying the APIs (essentially). For articles and news, we both aggregate organically discovered articles from streams, as well as scrape 250k+ RSS feeds and other news sources. So, articles are stored local to the server (which is why we have such a large collection of them), but sensitive social information is a pass-thru to the social network.
However, in both cases, we're doing lots of analysis on top of the data per our "normalization" philosophy.
Who's behind this? There's not a single name anywhere on the .me or .com sites. It's hard to trust a platform service without knowing who is building it.
This is a nice announcement to see, especially your support of searchable article feeds. Quick question: I see in https://streamified.me/developer/#concepts that we can search news articles, is there a list of sites that you cover, or is it safe to assume that you crawl over all major news services (tech, politics, money, etc)?
Also like your support for ADN. I have an account there that is sitting empty, might build something up over the weekend.
We should cover all major news sources. We have at least 250,000 different sources we're scraping, plus we also "crowdsource" news discovery. That is: organically (stream-discovered) content gets indexed as well.
I try this with js sample app + twitter auth. I get Error: {"type":"Authentication","status":401,"details":{"message":"OAuth Authentication Failure: invalid oauth_token syntax"}}
In the other hand, this could be handy for my news reader app for ipad. This notify of new content? Is this suitable for manage several users, each one with their own rss subscriptions?
Sure, we both cover social login and sharing. However, Streamified supports a whole bunch of news features, and a lot more social endpoints, which Singly doesn't begin to touch on.
You lost me here: "We think that formats like RSS don't make sense for today's internet".
And honestly.. another API? So it only works as long as you are in business (which is based on what by the way?) or decide to kill or change it? And also posting to my social media will be going through your machines? Seriously?
If you'd offer me an opensource script that I could call like "localhost/post?message="hello"&where="twitter,blog,facebook".. that would be different story. But hey, good luck locking others in your new service.
RSS is full of embedded HTML. It's practically impossible to make a good news reader without doing some heavy manipulation on it and trying to extract/modify ugly elements. We have a full blog post on the topic here: http://blog.streamified.com/post/45846467833/the-future-of-s...
There's a discussion below this one why it makes sense to have a single API for this, instead of open-sourcing the code. The benefits are numerous (we're hooking into many paid APIs and covering the costs, for example).
I agree that it would be awesome if it was opensource so you could mitigate any risks by having it run on your own hardware, but really it's a valuable service and they deserve to make some money, which will play into keeping the service alive.
The stack which is required for the Streamified API is really quite large, too. Between node, mongo, elasticsearch, and a few other pieces of software... plus a few dozen node modules and even some custom linux configuration, it's really not feasible to expect devs to install it on their own machines ;)
One could run their own NewsBlur, but Samuel has stated it requires a lot of resources to pull RSS feeds, and so in practice it's much easier to pay them $2/month. (This may not apply if you plan on having business customers who are paying orders of magnitude more than that.) But it alleviates any fear that the service users depend on may disappear; in the worst case, run it yourself.
Fair points, thanks. I'd have to discuss with the team if this is an option, ultimately. I can promise that if we ever did go away, I would certainly aim to opensource it (why not?) But as a company w/ investors, it's not a decision I can make myself right now ;)
Your welcome! I get that the service your providing is more than the sum of its code. Someone is doing the website, marketing, documentation, fixing bugs, keeping up with changes etc... not to mention paying the bills to reliably run the servers.
If you want my opinion, I would recommend finding a small area of your tech stack that you wish was better and starting an opensource project to solve that specific problem. No one expects twitter to give away their source code for their core product, but they do appreciate bootstrap and bower.js a whole lot.
I would agree if we were talking about a SOAP protocol that could be restful or something. But RSS doesn't bother me at all. You can just pass it through a converter to suit your fancy, and the bloat for an average use case of 10 results can't be that much. It's a minor miracle anybody decided to publish their sites through RSS so I'm happy they used the format that was hot at the time and got adoption.
I love RSS, I couldn't have built a ton of cool stuff over the years without it. For example I built @twuoted using twitter's search feed 4 years ago with Sweetcron, but that has long since been replaced by twitters API. I don't hate on RSS, I think it was a wonderful invention for it's time and was great for showing how standards can work.
But I still want it to die and JSON to become the new king in data transfer. That could be due to my work being in the web development arena and using node.js, but I honestly think it's just a better solution.
Hey just wanted to say that I appreciate the API and it has very nice docs! Ignore the naysayers, it's not for everyone but I can already imagine a ton of uses.
I'm not following... how is this broken? The headers return a 302 code, pointing to the proper HTTPS site. If you click the link you pasted, your browser should properly redirect (tested in FF, Chrome).
Sorry, but this is a developer portal -- not a news source or blog -- and really isn't meant to be used on a mobile device. I acknowledge that this may not be the answer you're looking for, but we didn't design the site for mobile because what it needs to do isn't really appropriate (or feasible) on mobile -- eg, the API console.
Well it is a news source, called hacker _news_. You telling me a developer doesn't read stuff on a cell is just complete BS.
I never said i want a mobile site. Just a css that doesn't completely break on small resolutions. If you don't want to fix it fine. But don't give me such a BS answer.
I acknowledge this might not be the answer you were looking for.
I'm sorry, I think I misspoke. What I meant is: you're right. What we should have done, though, was link to a blog post for you to read, rather than the dev portal itself. It just wasn't realistic to build a mobile enabled dev portal, so we should have posted something mobile friendly instead.
45 comments
[ 2.9 ms ] story [ 108 ms ] threadHow can folks help build this out? Could it be an open service?
To your questions: one of the major benefits of the service is the "crowdsourced" aspect of it. That is: the more people using the same API, the more news articles get found and indexed. Furthermore, the API actually hooks into other paid APIs, so it actually wouldn't work very well for you to run it on your own servers unless you wanted to also pay for them. For all these reasons and more, we think that a hosted API is the best solution.
Yes, our searching capacities include everything from categories to sentiment to tags to media... you name it. ElasticSearch powers our backend, which is powered by Lucene, so anything in an article can be searched. Check out the articles/search endpoint: https://streamified.me/developer/#endpoint=GET%20articles%2F...
I'm curious, are you proxying apis or crawling and keeping the information local to serve?
For social networks, we're proxying the APIs (essentially). For articles and news, we both aggregate organically discovered articles from streams, as well as scrape 250k+ RSS feeds and other news sources. So, articles are stored local to the server (which is why we have such a large collection of them), but sensitive social information is a pass-thru to the social network.
However, in both cases, we're doing lots of analysis on top of the data per our "normalization" philosophy.
Also like your support for ADN. I have an account there that is sitting empty, might build something up over the weekend.
We should cover all major news sources. We have at least 250,000 different sources we're scraping, plus we also "crowdsource" news discovery. That is: organically (stream-discovered) content gets indexed as well.
Hope to hear from you! Zane@Streamified.com
This is far better, though.
In the other hand, this could be handy for my news reader app for ipad. This notify of new content? Is this suitable for manage several users, each one with their own rss subscriptions?
Would you mind shooting me an email to Zane@Streamified.com so we can discuss?
And yes, you could easily manage many users' RSS feeds :)
And honestly.. another API? So it only works as long as you are in business (which is based on what by the way?) or decide to kill or change it? And also posting to my social media will be going through your machines? Seriously?
If you'd offer me an opensource script that I could call like "localhost/post?message="hello"&where="twitter,blog,facebook".. that would be different story. But hey, good luck locking others in your new service.
There's a discussion below this one why it makes sense to have a single API for this, instead of open-sourcing the code. The benefits are numerous (we're hooking into many paid APIs and covering the costs, for example).
I agree that it would be awesome if it was opensource so you could mitigate any risks by having it run on your own hardware, but really it's a valuable service and they deserve to make some money, which will play into keeping the service alive.
The stack which is required for the Streamified API is really quite large, too. Between node, mongo, elasticsearch, and a few other pieces of software... plus a few dozen node modules and even some custom linux configuration, it's really not feasible to expect devs to install it on their own machines ;)
One could run their own NewsBlur, but Samuel has stated it requires a lot of resources to pull RSS feeds, and so in practice it's much easier to pay them $2/month. (This may not apply if you plan on having business customers who are paying orders of magnitude more than that.) But it alleviates any fear that the service users depend on may disappear; in the worst case, run it yourself.
If you want my opinion, I would recommend finding a small area of your tech stack that you wish was better and starting an opensource project to solve that specific problem. No one expects twitter to give away their source code for their core product, but they do appreciate bootstrap and bower.js a whole lot.
But I still want it to die and JSON to become the new king in data transfer. That could be due to my work being in the web development arena and using node.js, but I honestly think it's just a better solution.
I acknowledge this might not be the answer you were looking for.