(Submitter here) This is so frustrating. There are so many links to important datasets that are just broken, such as the NHTSA crash ratings: http://catalog.data.gov/dataset/new-car-assessment-program-n... (try clicking "download" for an empty XML file)
The others link to agencies' download pages. It feels like they forgot that the purpose of data.gov is data first.
I've been attempting to extract various bits of government data and found the process to be quite tedious. The fastest way has been to use google vs data.gov to find particular data sets.
Edit: Just add '-site:data.gov' to your google search to help find the actual data set ;-)
Something makes me feel like that might be an uphill battle. The focus of the entire site appears to be a slick superficial UI. There is no attempt to consolidate data from various agencies into a uniform access format (say a JSON API or periodic dumps to a user-pays S3 bucket). The data is in XML or CSV or even XLS. The whole implementation seems a bit misguided, to be honest.
Those clauses seem contradictory. You have a very clear (and in my opinion on point) critique of what you're presenting us, but you don't have the inclination to write a blog post? You even wrote your own top-level comment providing some context. That would have made a sufficient blog post right there.
(A great blog post might have compared bad vs good cases, or provided some suggestions on improving, or even looked more broadly at whether this was an outlier of badly structured/provided data or just the norm)
The whole "Try (downloading|accessing) this..." strikes me as the same kind of title as Upworthy's "You'll never believe this..." in that it seems vaguely like link-baiting. I'm a little too curious not to click just to see what the dataset is about.
There are mediums (pun not intended) where you can make the equivalent of a blog post without much effort (Medium, Quora, I'm sure there are others). And link-bait is more about the marketing used to get people to click on the link (e.g. editorializing), isn't it?
I don't want to get into semantics, so let's consider this a minor detail, because the more important matter is that you're editorializing. HN has a pretty clear (and helpful) set of guidelines on posting:
http://ycombinator.com/newsguidelines.html
>If you want to add initial commentary on the link, write a blog post about it and submit that instead.
I'm not trying to harangue you or anything; you make a good point but it's hamstrung by a length-limited title section.
I really appreciate your efforts in explaining the posting guidelines, here! It's about respecting the forum; this is _not_ some one individual's personal editorial-logspace, but instead a place to share and discuss full-bodied web content.
Now, to defend this poster -- it's a quick point that speaks volumes.
This is a problem close to my heart, thanks for pointing it out. I have a project up at http://commonwealth.io attempting the reduce the barrier to using data like this. Even when the data is accessible, it could be in any format and you have to load it into a DB to get any value.
The idea is to enable real SQL queries directly on data sets, so you don't need to worry about accessibility or how you'll query it. Not much data in it now, and it doesn't solve your immediate problem, but perhaps a better model for storing and representing this data.
This is a great example of a use case for using Bittorrent Sync. They just need to create a read-only secret for each dataset they want to share and then put it on the website. Then the data is P2P'd for cheaper and much easier to maintain. Plus if the Government shuts down or has an issue the data can still be transferred.
Thanks for pointing this out. We've updated the highlighted dataset to something that is better maintained, but if you find other problems please do ticket them on github (https://github.com/GSA/data.gov/issues)
We're also working on some basic automated tests to ensure listings have the correct MIME type (which is incorrect far too often) and that URLs aren't 404s or redirects or HTML pages rather than raw data.
26 comments
[ 3.0 ms ] story [ 13.8 ms ] threadThe others link to agencies' download pages. It feels like they forgot that the purpose of data.gov is data first.
Edit: Just add '-site:data.gov' to your google search to help find the actual data set ;-)
https://explore.data.gov/Science-and-Technology/DOE-Green-En...
But, I figured I'd try to download the CSV to make sure it works. No dice. I end up with this:
http://www.osti.gov/home/RETIRED/greenenergy?format=csv
"The DOE Green Energy product has been discontinued. All of the information in DOE Green Energy can be found in SciTech Connect."
OK, so I follow the link to SciTech Connect:
http://www.osti.gov/scitech/
Of course it is to a general landing page. So I search on that page for: "DOE Green Energy" (using quotes)
And I'm presented with one result: "Ellens Test2". Here's a screenshot: http://i.imgur.com/08rHfcZ.png
If that's what you're looking for, then I found it. If not, I give up.
Thank you for the effort :) Unfortunately, I have no idea what that is, but someone could probably tell Ellen that it works.
https://www.usajobs.gov/GetJob/ViewDetails/363324500
(A great blog post might have compared bad vs good cases, or provided some suggestions on improving, or even looked more broadly at whether this was an outlier of badly structured/provided data or just the norm)
The whole "Try (downloading|accessing) this..." strikes me as the same kind of title as Upworthy's "You'll never believe this..." in that it seems vaguely like link-baiting. I'm a little too curious not to click just to see what the dataset is about.
Correct. I don't have a blog, and don't care to set one up. And I'd imagine a .gov link would have a lower chance of being linkbait.
I don't want to get into semantics, so let's consider this a minor detail, because the more important matter is that you're editorializing. HN has a pretty clear (and helpful) set of guidelines on posting: http://ycombinator.com/newsguidelines.html
>If you want to add initial commentary on the link, write a blog post about it and submit that instead.
I'm not trying to harangue you or anything; you make a good point but it's hamstrung by a length-limited title section.
Now, to defend this poster -- it's a quick point that speaks volumes.
"A link is worth a thousand bytes," sometimes.
http://www.osti.gov/home/RETIRED/greenenergy?format=csv
Which leads to here:
http://www.osti.gov/home/ostiblog/osti-re-focusing-and-re-ba...
Which explains the data is here:
http://www.osti.gov/scitech/
Which I appreciate is not what you wanted - but it's what you got.
The idea is to enable real SQL queries directly on data sets, so you don't need to worry about accessibility or how you'll query it. Not much data in it now, and it doesn't solve your immediate problem, but perhaps a better model for storing and representing this data.
These kinds of issues will improve as a new approach to listing datasets is implemented. This is currently underway and is described on Project Open Data - http://project-open-data.github.io/implementation-guide/
We're also working on some basic automated tests to ensure listings have the correct MIME type (which is incorrect far too often) and that URLs aren't 404s or redirects or HTML pages rather than raw data.
For those interested, please do consider applying for the open position - https://www.usajobs.gov/GetJob/ViewDetails/363324500
I'll do my best to answer any questions about the position and the process of applying.
See http://project-open-data.github.io/schema/ for details