HN Explorer

Ask HN: How to rip videos from sites like CNN.com?

3 points by skcin7 ↗ HN
So, my brother is a film maker and I am a computer scientist and we had an idea the other day to make a documentary film about the 2012 US Presidential Election and the role that big business plays in it. Who knows if we will actually complete this. Right now it is just an idea.

What is the best way to rip videos from "streaming only" sites like CNN.com? An example of some footage I'd like to get is:

http://www.cnn.com/2011/POLITICS/09/07/republican.debate/index.html?hpt=hp_t1

I examined the page with a HTTP Debugger and also viewed source code but could not seem to find the original source URL of the video. I was wondering if it is possible to find the URL of the video somehow and if it is, how?

Thank in advance for any insight you may provide! :-)

Also, it has just crossed my mind that this type of question might not be acceptable here, since you're not really "supposed" to download videos from streaming only sites. If this type of question is not acceptable please politely let me know.

EDIT: I intend to proceed only after obtaining the copyright owners permission.

10 comments

[ 2.9 ms ] story [ 36.5 ms ] thread
rhizome
For a documentary film you're going to need copyright clearances, so you should start at the licensing office of CNN.
_delirium
Depending on the documentary, you can often include segments of news footage for the purpose of commentary, especially if your commentary is on the news commentary itself (e.g. criticizing CNN's coverage). That's one of the main purposes of fair use, since otherwise CNN would be able to veto critical commentary of its broadcasts.

See, e.g., the Daily Show, which does not license the news footage it shows.

skcin7
I will have to look into that. Thanks for the response. Does anybody know how to rip videos from various websites, though? I feel like it is definitely possible. I use a program called JDownloader which can rip videos from Youtube, MegaUpload, and various other "streaming only" sites. Somehow the program is able to extract the URL of the video file(s) and download them directly. This program however does not support any of the news sites (including CNN).

At this point I am mostly curious about the theory. I am curious how a hacker would go about finding the URL and download the files themselves. I looked at Youtube source code and examined it with a HTTP debugger and can't seem to figure out the URL of where the video is stored but I know it is possible because the JDownloader program does it. I am sure that I, along with many here, would be able to benefit from knowing how to download from various "streaming only" video websites.

rhizome
I think HN is not a good place for this question, then. Good luck.
acak
I think you ought to make it clear that you intend to proceed only after obtaining the copyright owners permission. Might stop people from flagging it.
skcin7
Good point. I agree with your suggestion and have made an edit to my post.
ChrisNorstrom
I'm not a lawyer but...

Colbert and John Stewart do it all the time and I doubt they ask for permission for every single little clip. It would count as 'fair use'.

http://www.orbitdownloader.com/

Use Orbit Downloader, it rips everything, I've been using it for a year or two now, it's amazing. Open up Grab++ within Orbit, reload the page and the video will show up in Grab++, select it, hit download and it's yours.

skcin7
Hmmm very interesting. First of all thanks for posting that, it works perfectly as well, using the Grab++ tool.

The method that Skalman posted using the Firefox Web Console works perfectly as well, although in the CNN video that I posted above, that method seems to download a .flv file, whereas your method seems to be downloading a .mov file. Interesting....

EDIT: Both the .flv and the .mov seem to be the same file. They are both exactly 11,840,167 bytes, so that leads me to believe that they are the same file. It is interesting however that your method and Skalman's method download the same exact file just with a different file extension. I wonder why this is.

Skalman
Using Firefox Web Console it's possible to obtain all the fetched URLs. The one containing the video was: http://ht.cdn.turner.com/cnn/big//us/2011/09/07/bts-msnbc-pe...

I don't have a generic way of extracting this information from any site, but looking at those network sources that are used is always a good place to start. Using extensions such as Flashblock will let you control when you're going to start downloading, and there will be fewer URLs to look through.

skcin7
Oh hell yes. This method works perfectly :-)