I'm not too familiar with WebSockets. How trivial would it be processing power wise to provide this service? How trivial would it be for a large news organization like BBC to do something like this?
trivial processing power but news organizations tend to want traffic on their websites where you can see ads and click on them and other content. Not sure for how long they will even support RSS let alone make an effort to support websockets. They begrudgingly push stuff into social media only because it significantly pushes traffic to their sites.
The Times had links available with comically out of date (years) feeds - I can't find the page now, so maybe they did respond to my tweet in a way... although not quite I'd hoped for if so!
Hi, the pricing depends on the feature set that the clients require (articles/events/trends, amount of data, frequency of access) - that is why we don't offer it directly on the site.
Also, you are able to create an account for free (without CC) and make 2000 searches for free.
Really great idea, thanks for sharing! I do wonder, is this really annotated feeds?
For me, this is machine-curated feeds rather than a form of annotation given no additional information (besides meta-data scrapped and categorised) is displayed. Not quite as catchy, but annotations make me (at least) think of something else.
Note: am passionated about designing to support annotations on media directly.
Afaik there is no meta data scrapped (technically I think some HTML meta tags are scrapped but not sure if used).
There is machine-learning in the annotations - categories rely on a (cross lingual) text classifier, entities rely on matching to Wikipedia articles, maybe a bunch of small other things to - I don't know all the details - there are some papers published about it.
The original crawler (newsfeed.ijs.si) paper is from
Trampus, Mitja and Novak, Blaz: The Internals Of An Aggregated Web News Feed. Proceedings of 15th Multiconference on Information Society 2012 (IS-2012).
I got access for free. You should really ask them (they are responsive) I think it depends on org type/size - but the figures I heard unofficially were in the netflix (per user) price range.
I used it only for research (1 for media bias analysis related to the refugee crisis and 1 at the intersection of social/media and economy).
I know the people who started EventRegistry (same department/institute as me) so I didn't care so much about finding alternatives (I.e I got free access, I think most research users do have free access)
Data quality was mostly ok but some artifacts of the article content extraction (e.g. Something like readability) were present like bits of social media share buttons. Also some updated articles were not marked as updates but rather as new articles (after some threshold of changes).
The API has been improved since I last used it - but I found it a bit unintuitive and complicated at the time (almost a year ago). From my understanding it is a bit engineered in order to prevent users from accidentally doing expensive queries or pulling in too much data. kind of makes it harder to use. But still simple - you get up and running Withings minutes and just refine your queries as you go. API limits were generous and you could always resume work after the limit reset (I was getting historical data rather than following the stream)
We are also specifically targeting investing, as opposed to a more broad market (for now).
Piglet, tracks top news stories, identifies how domain experts feel about said news stories (as well as general topics), and provides trend, sentiment, and net promoter tracking. We just launched a very rough MVP a few days ago.
If you're interested, on the first of every month, we send a survey. If completed, you will receive the next month free of charge. It's free as long as you keep providing feedback.
Yup, only handle news, sentiment, and trends on subject matter.
It's basically an engine bubble up only the most relevant news. Then get experts opinions on said topics / news.
We are also very close to releasing our financial advisor, based on top of the data. Currently, it's written in python, so we need to make it accessible via the website.
Cool - I didn't know this project. I might take your offer on the free with survey next time I work on a research project with economists or finance people. (If the offer is still available)
In a eU research project we used this http://eventregistry.org/correlate
+ regression to predict oil barrel prices. We got better initial results than an information market from our partners but project ended and I'm not sure we'll get around to a more thorough analysis and publication of results.
28 comments
[ 3.0 ms ] story [ 73.2 ms ] threadRelated: it would be cool if there was a directory of public WebSockets offering a variety of real-time data streams.
Many don't. Examples I tried recently include:
The Telegraph (dead link from http://www.telegraph.co.uk/topics/follow-us/ and feeds available for some sections at http://www.telegraph.co.uk/{section}/rss.xml - not e.g. politics)
The Times had links available with comically out of date (years) feeds - I can't find the page now, so maybe they did respond to my tweet in a way... although not quite I'd hoped for if so!
I don't see any free public API like twitters mini firehose (10% of traffic for free).
For me, this is machine-curated feeds rather than a form of annotation given no additional information (besides meta-data scrapped and categorised) is displayed. Not quite as catchy, but annotations make me (at least) think of something else.
Note: am passionated about designing to support annotations on media directly.
There is machine-learning in the annotations - categories rely on a (cross lingual) text classifier, entities rely on matching to Wikipedia articles, maybe a bunch of small other things to - I don't know all the details - there are some papers published about it.
Event Registry – Learning About World Events From News http://wwwconference.org/proceedings/www2014/companion/p107....
Using news articles for real-time cross-lingual event detection and filtering http://ai2-s2-pdfs.s3.amazonaws.com/f917/c0cff24fed1af45f94c...
https://scholar.google.co.uk/citations?user=5pAxBWsAAAAJ&hl=...
The original crawler (newsfeed.ijs.si) paper is from
Trampus, Mitja and Novak, Blaz: The Internals Of An Aggregated Web News Feed. Proceedings of 15th Multiconference on Information Society 2012 (IS-2012).
- What did you use it for?
- What where the alternatives you considered?
- What was your overall experience? (eg. data quality or API ease of use)
I know the people who started EventRegistry (same department/institute as me) so I didn't care so much about finding alternatives (I.e I got free access, I think most research users do have free access)
Data quality was mostly ok but some artifacts of the article content extraction (e.g. Something like readability) were present like bits of social media share buttons. Also some updated articles were not marked as updates but rather as new articles (after some threshold of changes).
The API has been improved since I last used it - but I found it a bit unintuitive and complicated at the time (almost a year ago). From my understanding it is a bit engineered in order to prevent users from accidentally doing expensive queries or pulling in too much data. kind of makes it harder to use. But still simple - you get up and running Withings minutes and just refine your queries as you go. API limits were generous and you could always resume work after the limit reset (I was getting historical data rather than following the stream)
https://projectpiglet.com/
We are also specifically targeting investing, as opposed to a more broad market (for now).
Piglet, tracks top news stories, identifies how domain experts feel about said news stories (as well as general topics), and provides trend, sentiment, and net promoter tracking. We just launched a very rough MVP a few days ago.
If you're interested, on the first of every month, we send a survey. If completed, you will receive the next month free of charge. It's free as long as you keep providing feedback.
You only handle the news?
It's basically an engine bubble up only the most relevant news. Then get experts opinions on said topics / news.
We are also very close to releasing our financial advisor, based on top of the data. Currently, it's written in python, so we need to make it accessible via the website.
In a eU research project we used this http://eventregistry.org/correlate + regression to predict oil barrel prices. We got better initial results than an information market from our partners but project ended and I'm not sure we'll get around to a more thorough analysis and publication of results.
Good luck - I like your idea.
I'm curious to see this in action but it seems like I have to pay. Any idea what they could be doing?
Feeding tags and dates into a machine learning model to predict future events?