very cool. I've been thinking of building a distributed analytics service like this -- can you talk about what the architecture of something like this is? If I had to build a custom chart, I would have to go get my own data?
It would be awesome to have a service that provides hosted data, allows anyone to make charts / random transformations / add extra data and then add that to the main dashboard.
I'll leave the architecture of the cloudant db to the Cloudant guys. Being able to add data, though, is interesting. If people want to add different views (which basically trasform the data using map/reduce), we can add those to the db so they are available to all.
For charts, you just create a new chart and assuming it works and all, we'll host it on the site. It will pull the data directly from the db via the REST interface.
So, basically, yeah - we are hoping to do exactly what you are asking.
In our spare time, we're researching this dataset in detail. Here are some questions that we're interested in. Would love to hear other ideas and to have folks dig into the data. I think this dataset may be of interest to hackers, researchers and marketers.
1. Are the trajectories (e.g. rank vs time) for all popular posts of the same shape? They look ~logarithmic.
2. Are there identifiable clusters when you look in 4d space for rank vs points vs comments?
3. How does the impact of a post depend quantitatively on its respective cohort. I.e., what's a good model to normalize performance based on what else was happening that day?
4. What fraction of posts have comment threads that are "hijacked" by the first comment? Is their a quantitative way to find this, perhaps by looking at (2) above?
5. What are more detailed metrics to collapse "performance" of a post onto a single number?
6. How does performance on HN compare to reddit, etc?
7. How is the HN community different than other communities, if at all?
8. Given the time-dependent data, can we create a good estimator for the number of active HN users per day? Or can we at least create a relative ranking of the number of unique users between different days?
# of comments vs points would be interesting, and if you are willing to crawl the comments then diversity or complexity of commentary would also be interestin g- eg several multi-threaded discussions vs. a string of 'this is awesome' comments on some popular but shallow topic (eg Huble space telescope imagery or somesuch, which tends to attract much admiration but not necessarily a lot of discussion).
I'll probably be down-voted to hell, but whats with all the redirects that make "back"ing out of the site back to HN so painful, I counted three redirects before I landed on the final page, and had to long-click my back button to avoid the obnoxious redirect trap. Looks like your tracking (I assume) might drive people away.
Ah - shoot - that's probably a recently introduced bug! My bad. I wanted to include the chart of this Show HN post itself on the home page (as an example), and that is apparently causing all sorts of weirdness. I'll try to figure it out. Sorry.
Fixed. I had to remove my cool embedded, recursive chart showing how the post itself was doing. But yeah, that redirect back loop was maddening. So now to see the chart, you have to click on the Post History page or this http://www.hind-cite.com/multiPost?postIds=8075216
That's pretty sweet! Shameless plug: I built something just for your personal points a while ago using the Agolia API. It's not that sophisticated and detailed but it's good enough for my personal usage.
i) how many different people post URLs from a particular domain?
ii) how many different domains does a particular person post?
There's also iii but I'm not sure how to word it. It's something like "given a particular domain, what's the average[1] number of different domains posted by people who've posted this domain at some point?"
All of these questions, I believe, can be easily answered with the existing, Algolia-based, HN API. (Though good questions all.) What hind-cite adds is a time-based component. (The existing api shows the end result - number of points, comments, etc. etc.) Just an FYI.
I think marrying the two datasets (probably at a client level with two separate calls) would give a pretty complete picture.
Just a bit more generic. Cloudant sponsored this, so I wanted the db calls to be simple templates anyone could learn from (and everyone uses jquery, while angular is less common).
23 comments
[ 21.9 ms ] story [ 1065 ms ] threadIt would be awesome to have a service that provides hosted data, allows anyone to make charts / random transformations / add extra data and then add that to the main dashboard.
For charts, you just create a new chart and assuming it works and all, we'll host it on the site. It will pull the data directly from the db via the REST interface.
So, basically, yeah - we are hoping to do exactly what you are asking.
1. Are the trajectories (e.g. rank vs time) for all popular posts of the same shape? They look ~logarithmic.
2. Are there identifiable clusters when you look in 4d space for rank vs points vs comments?
3. How does the impact of a post depend quantitatively on its respective cohort. I.e., what's a good model to normalize performance based on what else was happening that day?
4. What fraction of posts have comment threads that are "hijacked" by the first comment? Is their a quantitative way to find this, perhaps by looking at (2) above?
5. What are more detailed metrics to collapse "performance" of a post onto a single number?
6. How does performance on HN compare to reddit, etc?
7. How is the HN community different than other communities, if at all?
8. Given the time-dependent data, can we create a good estimator for the number of active HN users per day? Or can we at least create a relative ranking of the number of unique users between different days?
Also, frequent violators of 'hijacking' the most popular comment by commenting on it. :)
https://hn.notmyhostna.me/
[0] http://www.chartjs.org/
[1] http://designmodo.github.io/Flat-UI/
i) how many different people post URLs from a particular domain?
ii) how many different domains does a particular person post?
There's also iii but I'm not sure how to word it. It's something like "given a particular domain, what's the average[1] number of different domains posted by people who've posted this domain at some point?"
I think marrying the two datasets (probably at a client level with two separate calls) would give a pretty complete picture.