- Is this more, less, or equivalently legal (copyright-wise) to just posting the youtube videos on youtube in the first place? It seems like the act of compositing them in this way would be significant, but maybe not?
- The normalisation of audio could use some work. SwitchCam seems to do something (the youtube volume on each clip was different for me), but it didn't quite work.
- I'd be really interested to know how automated this is, and how much human curation is required to get it right.
- Presumably, the next step is stitching the multiple videos together to make 3d models, allowing you to pan to places between the various camera operators. :)
Courts ruling on fair use are supposed to take into account "the amount and substantiality of the portion used in relation to the copyrighted work as a whole". Automated copyright detection schemes in YouTube may try to do some semblance of fair use by allowing snippets of copyrighted content under 30 seconds.
Yes it does. Just because the recording was permitted doesn't mean that the recorder has the rights to publish or replay the recording.
Another example: owning a DVD does not mean you can then stream the DVD over Justin.tv.
The thing about rights is that infringement must be claimed by the rightsholder for any action to be taken. Once a rightsholder claims infringement then the content/stream/video must be shut down or removed, and if the site or person continues to infringe, THEN action can be taken.
This is one of the good things about the DMCA and the reason YouTube, Soundcloud and even justin.tv can continue to operate. It is also exactly what the much maligned SOPA bill is trying to change - for the worse.
btw, you should sign a petition against that bill on votizen or similar.
They climbin in yo' windows, snatching your youtubes up.
just a sidenote - if bittorrent software ensure that no peer-to-peer interaction exceeds 30 sec. for any given title, would MPAA/RIAA/etc... still be able to claim "illegal dowload"?
When they take into account fair use, they take the entire case into account, rather than following any hard and fast rules saying that using X% is "fair."
In other words, no, there's pretty much no way they'd buy any attempt to game the system so transparently. That's why you need a lawyer.
No, the courts take context into account. So for example, it might be perfectly legal to post 2 sentences from a book on your blog, if 10,000 blog posters conspired to post a different 2 sentences to share the whole book, that would be clearly illegal. Furthermore, the law is interpreted by Judges, who are people, not machines, and they tend to take a very harsh stance against people who try to game the system.
- I'm no lawyer, but the advice we've received is that we're good on the legal side.
- You're on the money re: audio. It's miles better than it was 1 month ago, but we're still not 100% yet.
- There is a human curation step (~5% of the process), but that is mostly for removing really crappy videos. Searching/Synching/Sequencing is automatic.
- I dunno about next step(!), but i agree the idea is super interesting :)
You may be on the legal side, but I'd argue so were Amazon and Google when it came to letting the users stream their own music from their own "cloud" accounts. That didn't seem to stop the music labels from "demanding" getting paid again for the streaming, too. So you should watch your back and don't give in to them, especially if you know you're on the legal side.
Maybe this is something that has been mentioned before, but I would love it if I could select a fixed audio from one of the videos. For most videos theres one audio track thats clearly the best and it would be great if I could listen to that one the whole time while switching camera views.
PS: Very cool stuff!
> There is a human curation step (~5% of the process), but that is mostly for removing really crappy videos.
Actually, I'd be tempted to make the process 100% automated, and open it up to the general public as soon as possible.
This has the potential to go extremely viral, and occasional "bad" videos could work in favour of this, by introducing an element of humour into the proceedings.
I wonder how possible it would be both technically and legally to use the audio from the various angles as inputs into an audio processing step where you distill just the true music audio and clean it up. Ideally you'd have one 'cleaned' version of the audio playing and switching views just switches video.
This may be more legally grey than just piping through audio since you are actually producing a derivative work (even if it is mechanically produced), but the net effect would be awesome. You'd end up with better audio than any one person could record, and the more angles you get the higher quality you can make the audio.
Definitely a great idea and very well done for the first cut. You just need some Wilco on there.
It would probably be technically "possible" but I doubt that you'd end up with anything that sounds listenable.
The only bootleg audio that usually sounds decent are soundboard recordings, and when you find audience tapes spliced in with them the difference is immediately noticable.
I think theoretically you can make it quite good, but it's not going to be a simple task and you probably can't rely on purely the fans' recording. I think the best way would be to modulate them together based on a weighting depending on position or distance away from where the music is coming from, also discarding distorted recordings etc. That said, you probably only want this lightly mixed in along with a good recording from the desk, all adjusted for the viewing angle.
This certainly looks like a tortuous infringement in the UK (and I'd posit Europe). I didn't get a "only in the US" notice on YouTube though.
I can't see how this is possibly "fair use" - it's the complete work of music and the visual design of the set, choreography and show that is being reproduced in full in a commercial way. Unless the uploaders bought a license with their ticket to reproduce and distribute online and allow derivative works of those reproductions ...
Would be fascinated to read the letter from your IP lawyer justifying this?
>Supposed rights holders can easily contact Youtube with a DMCA request and they will take it down. //
This is largely irrelevant to the question of infringement and puts the onus on the owner to spot those infringing. Just because there's a ready way in which you can complain doesn't mean that the unlawful activity is somehow made lawful.
Note I'm making no comment here wrt the soundness or morality of said law.
I am excited about the concept - was thinking about this very idea while watching Coachella streaming live this year and then seeing all the cams people were uploading.
this is awesome. seriously. maybe you could package the technology in a way that lets other platforms use this without youtube. i am reminded of the original Color app.
Very cool, just a slight niggle about the 'lightroom' colour scheme (dark area surrounding the video) which I see being poorly implemented time and time again: it doesn't really work when you have elements of bright white on the page, because they just end up looking even brighter next to the dark parts. The glaring white strips are distracting while watching the video, moreso than even an all white background.
TL;DR: lose the strips of white from the video viewing page.
Thanks for the feedback, I agree actually. We've been talking about changing the color scheme to either be all dark or all white but haven't pulled the trigger.
Does anyone believe that this is done algorithmically? All the examples I checked seemed to be really poorly sync-ed, & as a DSP engineer I don't believe this task to be incredibly difficult...
would also be cooler to have a seating chart to click around to make sense of perspectives
It's all algorithmic but the flash player only seeks to the nearest key frame (every 2s) so that's why it isn't flawless. This can be overcome ... but I'd prefer to work on the seating chart thing - thanks!
This is one of the coolest things I have seen. While there are some little bits of feedback, I can admit that I spent the last 30 minutes watchings concerts, amazed at how great this felt.
MPAA: More of this, less of SOPA. If you give us great ways to get content, we will pay for it.
Sorry about that. It only happens when the JS borks out completely (which is rare), when JS is turned off, or when you're using a really old browser (which i assume you're not).
If you're inclined (I completely understand if not) can you tell us the error that the console in Chrome or Firebug gives you?
Thank you! I think you must be right about the adblock thing. We once had a similar thing with a Chrome plugin that borked things. We'll look into it; thanks again.
Interesting, thanks for poking around! I don't think we use anything dodgy (except maybe Facebook libraries) but I'll see if I can set up that configuration and debug it. Really appreciate you taking the time to write this up. Cheers, Chris
I'm having the same problem. I think it's a bad idea to use Facebook libraries for critical functionality. A lot of people (myself included) block Facebook resources from loading on external sites to decrease their ability to track user activity.
Cool idea, but I was sad that it wasn't finally an implementation of an idea I've had for a long time. My idea is to actually use video and audio information from distinct sources to create a single video/audio stream that is of better quality and/or completeness than any of the constituent parts. Essentially, my idea would do to video what Photosynth does for photos.
That is just so much harder; though I haven't looked at the website yet, seems to be overwhelmed right now. Besides, I would want one consistent audio signal instead of one that varies in noise, volume, whatever. Video from different sources is alright though, since we are used to switching scenes and cameras all the time.
The idea I'm talking about should provide a single consistent audio signal. I know nothing about audio processing, but it seems like it should be possible to take multiple bad audio signals and combine them into one signal that's better than any constituent audio source. Perhaps one audio source captured low frequencies well, while another captured higher frequencies better.
You can certainly automate crossfaded audio between multiple sources to try to get the cleanest copy, but it's hard. For instance, how do you decide whether it's noise or the letter "s" or the "chk" of a pick across muted guitar strings? The heuristics for "better than any constitutent audio source" can be extremely nuanced, algorithmically intensive, and still difficult to pin down, akin to speech recognition. Speaking purely to SaaS'y automated purposes, natch.
Typically what it seems you're talking about for audio here is similar to a matrix mix in the amateur/live audio world. People have been (manually) mixing soundboard audio with audience-recorded audio to improve the audio quality of recorded shows for some years now.
I don't know anything about audio processing algorithms, but (assuming there's a way), presumably with enough audio sources, there'd be a commonality between each one that describes the 'correct' sound. I.e. if there's different noise going on in each source (people talking around each microphone, at a gig, intermittently), you don't really need to decide which sound is 'clean' because you'd know which sounds are inconsistent...(?)
Sure, but it's determining what is "correct" that is the hard part. You could use a majority-rule if you have three or more sources, but the more additional sources are required starts getting into pretty niche territory and it still remains possible that the minority source is the most faithful one.
This seems unlikely to be possible. If you haven't captured any frequencies above 15kHz (which an average cell phone mic is unlikely to do), no amount of averaging, filtering, or combining will get them back. There will also be a considerable amount of distortion, since concerts tend to be so loud that even one's ears are distorting. Good luck separating physical distortion in the mic, limiter distortion in the analog or DSP stage, and clipping distortion at the ADC.
I think the best you could do is use the video to determine where someone was standing, and try to reconstruct some of the stereo information based on multiple recorders.
If you haven't captured any frequencies above 15kHz (which an average cell phone mic is unlikely to do), no amount of averaging, filtering, or combining will get them back.
I think this is technically not quite true. If two cell phones right next to each other are both sampling at 15kHz, in the best case you could combine their samples to get an equivalent sampling of 30 kHz. (Best case meaning phone 1 samples exactly half way between phone 2's samples.)
In practice, however, you would have to account for positioning and the fact that the phones' samples aren't perfectly offset from one another. It would require an amazing engineering feat to overcome this challenge, but I think it's within the realm of physically possible.
This is unfortunately so unlikely to as to be practically impossible (currently!).
If the microphones, ADCs etc on both phones are incapable of capturing frequencies of above e.g. 15Khz below a certain range, combining those signals definitely won't bring you any closer to the original signal. You may be able to cancel out a fair bit of noise given enough processing but you won't get back what hasn't been originally captured by either device.
That's before you get into phase problems from trying to combine two signals. A likely outcome is that the amplitude of some signals are increased whilst some are decreased due to phasing issues.
/fuzzily remembered music tech degree. May be too fuzzy though!
Sorry to be off topic here... but this is why I don't understand HN sometimes - the post above isn't nasty, augmentative, hurtful or 'bad' in any way, but instead of people responding to the poster they've downvoted him.
Isn't downvoting for removing bad content, not trying to silence someone you don't agree with?
Thoughts/comments?
(edit - post is no longer showing as greyed out/downvoted - but still, any comments?)
There will always be idiots who downvote great comments. I have on occasion been such an idiot myself (it happens). I wouldn’t let me bother by that, it usually all works out in the end. (Remember: It can take only a single person to grey out a comment.)
15khz is actually is very high frequency, most adults can barely hear it. The audio from phones is probably much more band limited than that. But that's the least of the challenges. People listen to and enjoy highly band limited music all the time: laptop speakers might have a frequency response of 500hz - 4khz.
The more challenging problem is the distortion from the phones being overloaded, crowd noise, built in limiters, different sample rates and compression. It is true that phase relationships from a single sound captured by multple sources can be very problematic.
However, the further the mics are away from the source the less this a problem at least with "phaseyness." This annoying artifact is a type of comb filtering , and it's based on the fact that two mics close to a sound source can be thought of really capturing the "same" sound at slightly different times. If the mics are far apart, the sound is no longer the same: it's picking up reflections from a myriad of sources, the phase relationships within the frequency spectrum have been smeared and shifted by traveling through air. This negates a lot of phase problems. The more likely problem is cancellation in the low frequencies which can be ameliorated with time alignment.
Only if the two phones are under-sampling the signal.
Chances are that there is a low pass filter in front of the phone's ADC, blocking signals above the Nyquist limit from reaching the sampler. Assuming brick wall filters (ie perfect cutoff), combining the signals will reduce variance (noise) but not give any information on frequencies above the cutoff frequency of the filter.
Brick wall filters don't exist though. What you might see is a miniscule amount of signal in the filter's stop band. Combining the signal from many many phones might reduce the variable enough to give useful information for frequencies a tiny bit above the cutoff frequency.
A cool project would be to gather the audio from every networked microphone in an area (mobile phones, laptops, ...) and use beam-forming techniques to reconstruct the sound pressure field as a function of position. My guess is that the system would be sensitive enough that it could do amazing things like capture conversations though walls or from long distances.
In some cases you will have a good prior on the clean signal from studio recordings. Of course, "registering" a recording (or parts of a recording) to the video would be a formidable task in itself.
If both channels have a similar spike at the same frequency at the same time, it is probably part of the signal (not noise), so combine those, and dampen all others. This would cover your case, if the other channel had enough of the low/hi freq of the other to relate them. I reckon Shannon looked at exactly this in developing Information Theory (for telephone signals on flaky lines), and it's probably all textbook stuff now.
The thing is, sound doesn't travel all that fast when you consider the wavelengths of vocal-range soundwaves. Those spikes are not going to arrive at the same time on the different phones.
As ever with DSP, phase problems will be the ruin of you.
nice point, but they'd synchronize with an offset. I doubt absolute time would be used to synchronize the videos anyway; they'd be matched by content.
Or do you mean that different frequencies will travel at different speeds, enough to make (e.g.) high and low frequencies arrive at different times? Whoa, apparently it does (http://en.wikipedia.org/wiki/Speed_of_sound#Effect_of_freque...) but seems to be a small effect.
>take multiple bad audio signals and combine them into one signal that's better
The problem eventually comes down to the fact that "better" is subjective. We're in the murky realm of art here. Should your algorithm keep that fret noise or the squeaking of a vocalist's intake of breath? Are they "noise," or are they part of the performance?
>I know nothing about audio processing
Not wishing to be rude, but this much is very evident. Recording engineers position their microphones with millimetre precision in order to combat phase issues, and that is in an ideal studio scenario. Doing what you suggest is basically impossible.
Maybe I'm overstating it, you could probably do something and it'd be a nice bit of research, but you wouldn't get useful results in the way that you're imagining.
I presume he means better in the sense that you'd try to remove overt noise, e.g. small conversations in the background, maybe wind noise. That sounds like it'd be possible to do. Improving a single video from multiple video sources would surely be impossible. Even with scene reconstruction etc you're not going to be improving the quality of any single video source...(?)
> Not wishing to be rude, but this much is very evident. Recording engineers position their microphones with millimetre precision in order to combat phase issues, and that is in an ideal studio scenario. Doing what you suggest is basically impossible.
Actually, that much I know, because I've done some amateur home recording. I know that, for example, when you mic a snare drum with two microphones that are pointed at each other, you have to put a phase inverter on one microphone. I also know my way around the basic processors for audio production (compressor, limiter, EQ, etc.).
What I don't know much about is the undoubtedly more advanced techniques which may or may not exist that could realize the idea I'm talking about. The best idea I can come up with is, if you had one audio source that captured the dynamics of a concert (perhaps from a phone that was far away from the house speakers), and another audio source that captured a clearer yet "smashed" sound (perhaps from a phone closer to the house speakers), perhaps you could apply a compressor to the second source that was keyed on the dynamics of the first. Again, I might be full of crap here.
We've actually thought about this, but it's a low yield proposition until we've shown there's a market for what we have now. It's not impossible, but it is very hard to do :)
I love it. Any chance you would expand to more general "news" applications? The audio wouldn't be as good for syncing against, but I've always wanted an interface like this for watching protest videos.
I would love this, too. Fortunately, it has a name that can become more general, too. But first focus on promoting your service as it is, to this "niche", and later you can try growing into more general stuff.
131 comments
[ 4.2 ms ] story [ 139 ms ] thread- Is this more, less, or equivalently legal (copyright-wise) to just posting the youtube videos on youtube in the first place? It seems like the act of compositing them in this way would be significant, but maybe not?
- The normalisation of audio could use some work. SwitchCam seems to do something (the youtube volume on each clip was different for me), but it didn't quite work.
- I'd be really interested to know how automated this is, and how much human curation is required to get it right.
- Presumably, the next step is stitching the multiple videos together to make 3d models, allowing you to pan to places between the various camera operators. :)
Another example: owning a DVD does not mean you can then stream the DVD over Justin.tv.
The thing about rights is that infringement must be claimed by the rightsholder for any action to be taken. Once a rightsholder claims infringement then the content/stream/video must be shut down or removed, and if the site or person continues to infringe, THEN action can be taken.
This is one of the good things about the DMCA and the reason YouTube, Soundcloud and even justin.tv can continue to operate. It is also exactly what the much maligned SOPA bill is trying to change - for the worse.
btw, you should sign a petition against that bill on votizen or similar.
They climbin in yo' windows, snatching your youtubes up.
In other words, no, there's pretty much no way they'd buy any attempt to game the system so transparently. That's why you need a lawyer.
- I'm no lawyer, but the advice we've received is that we're good on the legal side.
- You're on the money re: audio. It's miles better than it was 1 month ago, but we're still not 100% yet.
- There is a human curation step (~5% of the process), but that is mostly for removing really crappy videos. Searching/Synching/Sequencing is automatic.
- I dunno about next step(!), but i agree the idea is super interesting :)
Actually, I'd be tempted to make the process 100% automated, and open it up to the general public as soon as possible.
This has the potential to go extremely viral, and occasional "bad" videos could work in favour of this, by introducing an element of humour into the proceedings.
This may be more legally grey than just piping through audio since you are actually producing a derivative work (even if it is mechanically produced), but the net effect would be awesome. You'd end up with better audio than any one person could record, and the more angles you get the higher quality you can make the audio.
Definitely a great idea and very well done for the first cut. You just need some Wilco on there.
The only bootleg audio that usually sounds decent are soundboard recordings, and when you find audience tapes spliced in with them the difference is immediately noticable.
I can't see how this is possibly "fair use" - it's the complete work of music and the visual design of the set, choreography and show that is being reproduced in full in a commercial way. Unless the uploaders bought a license with their ticket to reproduce and distribute online and allow derivative works of those reproductions ...
Would be fascinated to read the letter from your IP lawyer justifying this?
What is the problem exactly?
This is largely irrelevant to the question of infringement and puts the onus on the owner to spot those infringing. Just because there's a ready way in which you can complain doesn't mean that the unlawful activity is somehow made lawful.
Note I'm making no comment here wrt the soundness or morality of said law.
I am excited about the concept - was thinking about this very idea while watching Coachella streaming live this year and then seeing all the cams people were uploading.
http://webcache.googleusercontent.com/search?q=cache:http://...
TL;DR: lose the strips of white from the video viewing page.
would also be cooler to have a seating chart to click around to make sense of perspectives
MPAA: More of this, less of SOPA. If you give us great ways to get content, we will pay for it.
http://en.wikipedia.org/wiki/Awesome;_I_Fuckin_Shot_That!
Incidentally, Switchcam.com was only a little more than $1k, which was definitely worth it.
If you're inclined (I completely understand if not) can you tell us the error that the console in Chrome or Firebug gives you?
VKLog is not defined http://switchcam.com/assets/js/utils/facebook.js?ver=2011112... Line 81
VKLog.dlog("checkSessionStatus()");
It does seem to work with Adblock disabled, so I'd guess a subscription is blocking something essential.
http://photosynth.net/
Typically what it seems you're talking about for audio here is similar to a matrix mix in the amateur/live audio world. People have been (manually) mixing soundboard audio with audience-recorded audio to improve the audio quality of recorded shows for some years now.
I think the best you could do is use the video to determine where someone was standing, and try to reconstruct some of the stereo information based on multiple recorders.
I think this is technically not quite true. If two cell phones right next to each other are both sampling at 15kHz, in the best case you could combine their samples to get an equivalent sampling of 30 kHz. (Best case meaning phone 1 samples exactly half way between phone 2's samples.)
In practice, however, you would have to account for positioning and the fact that the phones' samples aren't perfectly offset from one another. It would require an amazing engineering feat to overcome this challenge, but I think it's within the realm of physically possible.
If the microphones, ADCs etc on both phones are incapable of capturing frequencies of above e.g. 15Khz below a certain range, combining those signals definitely won't bring you any closer to the original signal. You may be able to cancel out a fair bit of noise given enough processing but you won't get back what hasn't been originally captured by either device.
That's before you get into phase problems from trying to combine two signals. A likely outcome is that the amplitude of some signals are increased whilst some are decreased due to phasing issues.
/fuzzily remembered music tech degree. May be too fuzzy though!
Isn't downvoting for removing bad content, not trying to silence someone you don't agree with?
Thoughts/comments?
(edit - post is no longer showing as greyed out/downvoted - but still, any comments?)
The more challenging problem is the distortion from the phones being overloaded, crowd noise, built in limiters, different sample rates and compression. It is true that phase relationships from a single sound captured by multple sources can be very problematic.
However, the further the mics are away from the source the less this a problem at least with "phaseyness." This annoying artifact is a type of comb filtering , and it's based on the fact that two mics close to a sound source can be thought of really capturing the "same" sound at slightly different times. If the mics are far apart, the sound is no longer the same: it's picking up reflections from a myriad of sources, the phase relationships within the frequency spectrum have been smeared and shifted by traveling through air. This negates a lot of phase problems. The more likely problem is cancellation in the low frequencies which can be ameliorated with time alignment.
Chances are that there is a low pass filter in front of the phone's ADC, blocking signals above the Nyquist limit from reaching the sampler. Assuming brick wall filters (ie perfect cutoff), combining the signals will reduce variance (noise) but not give any information on frequencies above the cutoff frequency of the filter.
Brick wall filters don't exist though. What you might see is a miniscule amount of signal in the filter's stop band. Combining the signal from many many phones might reduce the variable enough to give useful information for frequencies a tiny bit above the cutoff frequency.
A cool project would be to gather the audio from every networked microphone in an area (mobile phones, laptops, ...) and use beam-forming techniques to reconstruct the sound pressure field as a function of position. My guess is that the system would be sensitive enough that it could do amazing things like capture conversations though walls or from long distances.
The thing is, sound doesn't travel all that fast when you consider the wavelengths of vocal-range soundwaves. Those spikes are not going to arrive at the same time on the different phones.
As ever with DSP, phase problems will be the ruin of you.
Or do you mean that different frequencies will travel at different speeds, enough to make (e.g.) high and low frequencies arrive at different times? Whoa, apparently it does (http://en.wikipedia.org/wiki/Speed_of_sound#Effect_of_freque...) but seems to be a small effect.
Also, how do you calculate your offset? Consider that it is constantly changing.
The problem eventually comes down to the fact that "better" is subjective. We're in the murky realm of art here. Should your algorithm keep that fret noise or the squeaking of a vocalist's intake of breath? Are they "noise," or are they part of the performance?
>I know nothing about audio processing
Not wishing to be rude, but this much is very evident. Recording engineers position their microphones with millimetre precision in order to combat phase issues, and that is in an ideal studio scenario. Doing what you suggest is basically impossible.
Maybe I'm overstating it, you could probably do something and it'd be a nice bit of research, but you wouldn't get useful results in the way that you're imagining.
Actually, that much I know, because I've done some amateur home recording. I know that, for example, when you mic a snare drum with two microphones that are pointed at each other, you have to put a phase inverter on one microphone. I also know my way around the basic processors for audio production (compressor, limiter, EQ, etc.).
What I don't know much about is the undoubtedly more advanced techniques which may or may not exist that could realize the idea I'm talking about. The best idea I can come up with is, if you had one audio source that captured the dynamics of a concert (perhaps from a phone that was far away from the house speakers), and another audio source that captured a clearer yet "smashed" sound (perhaps from a phone closer to the house speakers), perhaps you could apply a compressor to the second source that was keyed on the dynamics of the first. Again, I might be full of crap here.
http://www.youtube.com/watch?v=xW4UVjROQdw
It combines a few high-resolution still images with a low-res video to improve the video quality:
Static scene: http://vimeo.com/1513129 Dynamic: http://vimeo.com/2937785
(Looks a bit old (3 years) so I don't know if it's still in development)