Wonderful technical achievement but I think I’d rather squint through garbled video to see a real human.
Now if I can use it to add a Klingon skull ridge and hollow eyes to my boss or scribble notes on my scrum master’s generous forehead we might be on to something.
I hope they use this kind of tech for the Shield TV, too, to make 360p/480p videos look like 1080p/4k in the next few years.
This would drastically set the Shield TV apart from any other "smart" TV or set top box. People would want to buy a Shield just for this feature alone. The DLSS for games is already (over) hyped, but this would make much more sense for videos and TV.
Depends on how big the network is. You can process neural networks on CPU with acceptable speed in certain cases. And nicer smart phones tend to have a decent GPU.
That would still have major advantages. You could use the extra bandwidth to stabilize everything. Even on the best of networks you have buffer pauses. And on wifi or cell you have a lot of those. If this can be used to improved those ... you'll have improved quality greatly.
Is it just me or do the videos no longer look natural? I feel like I see highly non-linear movements (parts of the video moving when they shouldn't, or vice-versa), and facial expressions don't really look quite the same.
fair, in that they aren't perfect photorealism, but if their comparisons with the regular codecs and techniques are correct, I'd take their modelled faces over the comparable- level of digital artifacts for the same bandwidth.
After all, it doesn't need to be said that when you're on a regular videoconferencing call and bandwidth starts to suffer, the resulting images don't really look anything like a photorealistic person either. I think this is actually a really good use of NN.
The thing is you only want to make this trade-off when the bandwidth is actually starting to suffer. It'd be nice if there was a nice way to make this adaptive and use NNs only when throughput is low, but the nonlinearity of the distortions makes me think this would be really hard. [1] I know what I don't want is for a normal conversation in an uncongested network to look unnatural or for facial expressions to get distorted unnecessarily.
Edit: [1] I meant to say doing a mixture of these (with the NN image as the "base", with H.264 to improve accuracy) seems really hard. On the other hand, just a hard switch from H.264 to NN when quality degrades is probably quite practical?
Adaptive use would be awesome. Especially on conference calls at work there is always that one (or ten) person whose connection is absolutely awful and looks like a giant pixelhead.
Perhaps it's just me, and my philosophical bent, but I can actually see coming at it from the exact opposite end.
I don't want bandwidth and things spent/wasted without it providing a significant benefit (I'm probably one of those 1080p/720p is good enough for most things type guys).
I definitely don't want work making large bandwidth or resource claims on my connection when I'm working at home. And if any of this remote working has taught me anything, it's that most of my colleagues don't have steady/reliable tech or connections, so i'd almost want it used pre-emptively as a default so we can spend the rest of those resources on robustness or other qualities. (I realise of course that at the moment none of them have high-grade Nvidia graphics cards, but I'm talking hypothetically in the far off future).
In short, I want a world where the cost to benefit ratio of things is orders of magnitude larger, because things like this let us spend network/resources on things which matter.
Yes, when I'm calling my parent/grandparent one on one I might want to upgrade the signal, but I don't need to see random colleague's face in all their HD glory, or remote people whom I have no idea who they look or sound like anyway (i believe that's also been one of the findings with deepfakes, that you don't notice the eerieness/falseness as much if it's a reference of a face that you don't have pre-determined knowledge of).
I tried to pay attention to what you and others flagged (parts not moving, facial expressions, shoulders, etc.) But I cannot spot anything out of place.
That said, looking at other comments here, you're definitely not the only one
This is perceptible to some degree, but in my opinion a highly practical way of applying this technology is to intentionally lower/shift the quality on the fly just a bit so as to "blend" the layer boundaries where the stitching happens.
That way the doctoring isn't apparent, you still benefit from the massive bandwidth savings (which I consider most important anyway), and it appears more believable in the real-world context of variable bitrates.
> they have managed to reduce the required bandwidth for a video call by an order of magnitude. In one example, the required data rate fell from 97.28 KB/frame to a measly 0.1165 KB/frame – a reduction to 0.1% of required bandwidth.
A nitpick, perhaps, but isn't that three orders of magnitude?
We've already seen people use outlandish backgrounds in calls, now it's going to be possible to design similar outlandish views, but actually be this new invention in real time. There's been a lot of discussion centered around deep fakes and its problems, this is essentially deep faking yourself into whatever you want.
Video calls are a very important form of communication at the moment, if this becomes as accepted as background modification, that would open the societal door to a whole range of self presentation that up till now was restricted to in game virtual characters.
I wonder what kind of implications that could have. Would people come to identify themselves strongly with a virtual avatar, perhaps stronger than their real life "avatar"? It is an awesome freedom to have, to remake yourself.
>Video calls are a very important form of communication at the moment, if this becomes as accepted as background modification, that would open the societal door to a whole range of self presentation that up till now was restricted to in game virtual characters.
Thanks for that comment. Can't wait for it to happen !
I think it’s an unnecessary nitpick. One statement is general and the other is a specific, extreme example. “usually 10x, sometimes 1000x”, that sort of thing
"An order-of-magnitude difference between two values is a factor of 10. For example, the mass of the planet Saturn is 95 times that of Earth, so Saturn is two orders of magnitude more massive than Earth."
> "More precisely, the order of magnitude of a number can be defined in terms of the common logarithm, usually as the integer part of the logarithm, obtained by truncation."
$ bc -l
l(835)/l(10)
2.92168647548360208478
That would make it 2 orders of magnitude by that method. Happy to accept that it's 3 orders of magnitude by the N=a*10^b method though. Either way, it's definitely not one.
> A nitpick, perhaps, but isn't that three orders of magnitude?
Perhaps the example was a best-case, and the usual improvement is about 10x. (That or 'order of magnitude' has gone the way of 'exponential' in popular use. I don't think I've noticed that elsewhere, though.)
I imagine people in home office situations would like to use this not only for the background, but them-self. I mean, if you are in doors all day, you might not be perfectly groomed for the day - so faking that would probably appeal to many people.
Today's meeting was an hour-and-a-half spent on bike-shedding the position--and color--of the "logout" button on our product page.
15 minutes were spent debating the resident usability expert who suggested white text on a dark blue background would be more readable for people with low vision. The department manager insisted on retaining pastel blue as it is his favorite color.
45 minutes were spent arguing over whether "logout" or "log out" comprises proper semantics. Our linguistic expert was unfortunately not able to attend as she was sent on a business trip earlier in the week.
The last 30 minutes were focused on team-building exercises as Bob doodled on his tablet and Susan smiled politely at the speaker whilst screaming internally as she had 3 more meetings to attend before her department could move forward.
45 minutes were spent by humans arguing over whether "logout" or "log out", while I created both buttons (GPT-3 can already do that) and A/B tested it.
My prediction is that people will just change their avatars as often as they change their personal fashion. For some that’s never and for others it’s every season or even more often.
sure is .. if you stream your face at >30-50Mbit/s. For contrast highest bitrate available on Twitch, used for streaming high motion full screen updating twitchy 1080@60 gaming, is ~6-8Mbit/s.
I think the ability to, as someone mentioned it, have yourself look a bit tidier than you actually are (working from home) could be a huge benifit.
I mean taking away focus on things that doesn't matter in a virtual meeting such as:
Where you are sitting - via Virtual Background
Your daily hair style status or if you have a nose pimple - Via NVIDIAs AI showcased here.
Would be great.
Though replacing yourself with a "digital" avatar I think takes away many of the benefits an actual live meeting provides.
It depends on how accurately the avatar is able to represent important information: emotion, attention, state of mind, etc. There's a lot of bandwidth in looking at a face (and bodylanguage as well), that's where the value in face-to-face meetings is.
3. I watch a lot of interview s with people in VR Chat and it's very interesting how people seem to find it easier(?) to open up while they are embodying a character. https://youtu.be/KZWOXgc7PA4
Being able to experiment with identity in this way is really interesting to me, and I hope it becomes more mainstream with the proliferation of this technology
There's a Webtoon (it's okay but not great) that had a premise I think will turn out to be precient and reminds me of your last bit. The gist is that it's a future where everyone wears vr goggles. As a result, the teens in the comic all have personalized visuals mapped to their bodies that you can see if you also have your goggle's on when you look at them. The cooler kids even have full blown avatars that cover up their entire body and make them look like everything from aliens to elves.
This sounds a lot like the movie Surrogates, where at one point the protagonist notices the badge of a android surrogate is completely different from the human behind the surrogate (as printed on the badge).
Jaron Lanier’s book on VR went in-depth on the importance of avatars, and experiences people had embodying different avatars — particularly in the early days of first-wave VR.
As these trends sort of become more and more prevalent, I am so shocked at how David Foster Wallace had nailed this prediction in his book Infinite Jest.
Humans becoming more and more dependent on virtual face-to-face meetings and also relying on embellishment of their supposed appearance through the screen. It reminds me of how SciFi authors predicted technology, but with a complimentary commentary on human psychology.
Sorry if it isn't directly related to the post, but it is so striking to me.
It's possible that the "order of magnitude" statement was the majority case, and the 0.1% statement was a best case scenario. So, 1 magnitude is to be expected, but 3 is possible.
This reminds me of a sci-fi novel I read in the nineties. The premise had something to do with actors who took on roles in virtual reality where their bodies are fit with sense-points. They're cast in live-action role-plays with wealthy remote clients. They're basically deep-fakes in VR.
Neal Stephenson's Diamond Age had an element like that. It was even possible that the actors ("ractors") didn't necessarily know what they were acting in, just the general parameters and the next set of lines and actions necessary to continue the performance.
David Foster Wallace predicted this in his novel Infinite Jest. Except they where static images inserted over a video phone, and the user had to keep their head positioned just right to make them work.
I wonder how weird it gets when you turn your head too much. This is very cool though - I was expecting to be able to tell a difference and maybe slip into uncanny valley territory but it looks good.
Big question though - is this just substituting the problem of not having good internet with not having a really fast nVidia graphics card?
If I understand correctly the sender just uses classic object detection/tracking. So the question would be how bad does it look if the receiver just tried to distort the image using that tracking data without having a trained model to smooth out the distortions.
Watching the video it no longer feels like you're looking at a real person, but instead just another npc. It no longer feels as personal. The last thing remote relationships need is more impersonality. I hope this is used only when it's needed.
I think it would be easier to see if they had a comparison between a high bit rate traditionally compressed video and their neural net compressed video. To me the facial shapes and movement seem just a little bit wrong like you’re looking at a 3D animation of a person instead of a video.
They do this using GANs so I don’t think it is compressed in any way. Instead, the points on the face are sent across the net and the face is actually generated. I’m sure it can get better with time. There is a site thispersondoesnotexist.com that uses the same tech but looks real.
> Instead, the points on the face are sent across the net and the face is actually generated.
This is not fundamentally different from other lousy compression algorithms.
The bits that are missing are reconstructed using a bias built in the compression algorithm. Deep learning based algorithms simply have a more realistic bias, so the artifacts that introduced to replace the missing bits are less noticeable.
I want to see if people notice this effect when they don't know it's artificially generated. I have a feeling that the uncanniness is at least part because you know it's fake.
How soon before this incorporates GPT3 and guesses what we were going to say anyway, so we no longer need to say it? Or doesn't quite guess right, and says something that gets you fired!
A technology very similar to this plays a plot point in Vernor Vinge's 1992 novel A Fire Upon the Deep.
In his universe, both the interstellar net and combat links between ships are low bandwidth. Hence, video is interpolated between sync frames or recreated from old footage. Vinge calls the resulting video "evocations".
I was thinking of exactly this when I read the article.
The plot point being that when the bandwidth gets too low, the interpolation AI has to make lots of stuff up, you are not quite sure exactly what was said.
I seem to remember the bandwidth in the book was very tiny, small number of bits per second (?) so the AI was taking the speech and compressing it into something more compressed than text then decompressing it at the other end into something that was more or less the same.
Wow, that’s a fascinating concept. Effectively a bitmap index of possible terms said and synthesized speech back on the other end based on old footage of how that person talks.
There is also a similar technoligy in Rob Reids After On book. The AI has the ability in thet book to "refocus" the person so that they are looking into the camera.
I believe this is huge and would create higher engagement if everybody was acutally looking into the camera instead of to the side or up all the time. Creating a more human an emotional relation with the people you are talking to.
I highly recommend A Fire Upon the Deep. Its a rare mix of really interesting hard scifi with an actually good story and characters. Hard scifi often has very flat characters but this is not a book which suffers from it.
It has a very very cool twist to explain the Fermi Paradox and is a really good example of a universe with one modified rule.
> Called “Free View,” this would allow someone who has a separate camera off-screen to seemingly keep eye contact with those on a video call.
Am I the only one who thinks eye contact on video calls feels creepy? I think I would prefer this feature to remove eye contact on video calls rather than add it.
Apparently researchers have measured this, and for people meeting in person it's normal to maintain eye contact 30%-60% of the time in a conversation [1] with each contact lasting ~3 seconds [2]
So a system that maintained eye contact continuously would indeed risk looking creepy!
Isn’t this just like apples animated emoji (Animoji) where your face is mapped to a emoji character? Except instead of a cartoon it’s mapped to your actual face.
Now the person you are speaking to is going to be n% (partially) emulated. n is going to increase in future. One day there will be a paid feature letting you emulate 100% of yourself to respond to video calls when you are not available. And finally, they will replace yourself even without you knowing, and even after you die.
That won't be possible unless full-brain emulation with live data sync gets invented.
Knowledge workers are all about the knowledge that they build up over time when working with a particular environment (be it an industry, a system, or a person/group of people). That knowledge is non-deterministically synthesised in the brain based on the experiences of that person, and being non-deterministic, no AI will come to the same conclusions about every item as this particular human would.
In that case, an emulated personality that is meant to make themselves available as your replacement will be an impostor. One that is less of an expert than yourself at best, and at worst one that is misinformed or misled on various issues (which in turn causes other people to be misled or misinformed).
Perhaps. In his Revelation Space universe, Alastair Reynolds categorises AI as Alphas (full brain simulations), Betas (non-conscious mimics which are based on a person’s behaviour over an unspecified but very long time period), and Gammas (which I read as being mere apps with voice interfaces).
If the goal is to make it seem like your present when you aren’t, I can believe we’re only a few years away from Reynolds‘ Betas — a Markov chain can’t mimic a human well, but it can mimic a human; GPT3 can do better, and while it still isn’t great, the main reason it feels like it might not be enough for public figures is how easy it is to get it to answer as if it were someone else rather than as The Right Honourable Sir Obvious Madeupname, MP for Oxbridge-upon-Wells who is paying for the chatbot to mimic him in particular.
This was part of what drove me off Twitter after the last election. I realized through my links and tweets that I had already provided enough training data for a bot that could not only sound like me, but share content from the future that I might be interested in.
Perhaps they are overdoing it (if you have a hammer, ...). I would think that the most useful way to use AI in this context would be to predict who is going to speak next.
This may be projecting expectations but the example compressed video looks very slightly fake in a way that is just a little uncanny valley type unsettling.
Perhaps the nets they’re using are compressing out facial microexpressions and when we see it, it seems just a little unnatural. Compression artifacts might be preferable because the information they’re missing is more obvious and less artificial. In other words i’d rather be presented with something obviously flawed than something i can’t quite tell what is wrong.
Fundamentally, I don't know if people realise that what we're on the verge of here.
It's effectively a motion-mapped keypoints of the person projected onto a simulated model. I'm assuming the cartoonish avatar was used as an example to partly avoid drawing direct lines to the full implications.
- There's no reason this couldn't extend to voice modelling as well. (much clearer speaking at much lower bandwidth)
- There's no reason this couldn't extend to replacing your sent projection with another image (or person)
- Professional looking suit wearing presentation when you're nude/hungover/unshaven. Hell, why even stop at using your real gender or visage? Imagine a job interview where every candidate, by definition, visually looked the same :)
- There's no reason you couldn't replace other people's avatar with one's of your own choosing as well.
- Why couldn't we model the rest of the environment?
Not there today, but this future is closer than many realise.
>>- Professional looking suit wearing presentation when you're nude/hungover/unshaven. Hell, why even stop at using your real gender or visage? Imagine a job interview where every candidate, by definition, visually looked the same :)<<
This would be interesting as an upgrade to the “name on resume” test.
Could also see a future company policy that runs peoples data through a “sameness” filter before letting them into the company to scrub bias.
TL;DR: You can't reconstruct a generic video stream from a dozen detected facial data points.
Not sure if I understood your post correctly, but you're slightly misled here.
Watch the video. This is not a new general purpose video codec, it is basically Deep Fake - taking a still image (key frame) and superimposing detected facial expression/movement on this key frame (leaving out some technical details).
This is an improvement over (not-anymore-)state-of-the-art h264 since transmission of only a few coordinates mapping your facial expression is significantly less data to transmit then a delta of arbitrary video and periodic keyframes (again generously leaving out important technical details of modern video codecs). Trying to reconstruct a moving car/background/etc. from these facial expression key points will lead nowhere.
But after that, I was reminded of the paranoia (or not?) around Zoom and that, for an extreme example, the CCP was mining and generating facial fingerprints and social networks using video calls. It seems like this technology is the same concept except put to a useful purpose.
If I had to guess, the issue around "black people" is that photos are 2D.
We don't really understand just how little information is actually in a photo (we add huge amounts of info in our perception).
My guess is that predictive systems are using contrast as a guide to essentially 3D structures which, simply, just cannot be reconstructed from 2D. And therefore, probably struggle more on dark faces which have different contrast properties.
While contrast is almost certainly part of it, I’d hazard a guess that the training set is also partly to blame.
Now, I don’t know much about neural networks (AI), but my understanding is that if you provide a training set representative of the population makeup, (in America at least) it’ll be biased towards white people as it hasn’t “seen” enough black person images. My limited understanding would then make me think one would need equal white person photos as well as black person photos.
Sure, but were those datasets of 3D snapshots of white people, it wouldnt matter.
Black faces and white faces are "statistically equivalent" in 3D.
The issue is more, in my view, the hubris of calling this system "facial recognition". It isnt: it's pixel pattern color recognition which sometimes coincides with certain facial patterns.
I imagine you're right on both counts. Contrast is certainly an issue. You definitely have to make adjustments to photograph dark skin (or animals with black fur).
Look neat. I wonder that the system requirements / license for software will be?
There's a real network effect with things like codecs - unless some significant proportion of calls can use it, it'll remain a cool but obscure experiment.
I hope Nvidia have the foresight to release something that'll run on any hardware, and under a permissive license, but I suspect not.
The idea is out there already (it's basically deep fake tech, right?), and I'm sure it won't be so long before some open source version of it gets released. Nvidia would be wise to get out in front of that and at least have their brand associated with a widely used variant on the theme.
You can get pretty good network speeds in space. Some Ka-band satellites (SpaceX Starlink for example) have gigabit networks between them. All you need to extend that out to Mars would be high gain directional antennas on things. If you used lasers instead it could be even higher bandwidth.
I have to disagree you there. I owned a satellite uplink/downlink in Afghanistan and it wasn’t cheap per megabit (supply/demand issues) and I highly doubt you’d be able to get a gigabit connection to Mars, especially when it’s on the other side of the sun. Also, the earth is rather noisy, so it may be easier to get a fast connection from Mars, the SNR from earth will be much lower.
I guess I'm also talking about the theoretical maximums in ideal conditions as well, which is sort of cheating on my part. There would be times when you can't get the optimal speed if, like you say, the sun is the way.
My tests already take twice as long to run when I’m on a Google hangout. For my case, I’d honestly rather use more bandwidth and do minimal local processing. If my machine is slowed down any more then I might have to stop working completely and focus on the meeting I’m in!
Modern machines aren't really designed for real computation like running tests or compiling stuff. They're made to render webpages and load up games. Run your computation tasks on a VM in the cloud...
Are remote development environments common?
There’s a CI server doing the heavy lifting but I still run tests constantly on a docker box while developing locally.
I develop entirely remotely. Save the files on a network mounted filesystem, run shell commands over SSH/mosh, etc.
The server is in the same city as me to avoid excessive latency.
The benefits of being able to develop anywhere (including from a phone in a pinch), and being able to add extra CPU's and RAM at the click of a button outweigh the need for a network connection for my usecases.
Admittedly I have no idea how resource-intensive this image processing is, but it's possible that it's less of a hit than the fact that all video call programs today run on HTML+JS. If Hangouts was a native program, there would be plenty of power to spare.
196 comments
[ 5.0 ms ] story [ 225 ms ] threadNow if I can use it to add a Klingon skull ridge and hollow eyes to my boss or scribble notes on my scrum master’s generous forehead we might be on to something.
This would drastically set the Shield TV apart from any other "smart" TV or set top box. People would want to buy a Shield just for this feature alone. The DLSS for games is already (over) hyped, but this would make much more sense for videos and TV.
[0] https://github.com/tensorflow/tfjs-models/tree/master/faceme...
This is a brilliant idea, even if the hardware is pricey today.
Disclaimer: no affiliation, but I use Zoom/Teams/Slack/FaceTime/YouTube.
After all, it doesn't need to be said that when you're on a regular videoconferencing call and bandwidth starts to suffer, the resulting images don't really look anything like a photorealistic person either. I think this is actually a really good use of NN.
Edit: [1] I meant to say doing a mixture of these (with the NN image as the "base", with H.264 to improve accuracy) seems really hard. On the other hand, just a hard switch from H.264 to NN when quality degrades is probably quite practical?
I don't want bandwidth and things spent/wasted without it providing a significant benefit (I'm probably one of those 1080p/720p is good enough for most things type guys).
I definitely don't want work making large bandwidth or resource claims on my connection when I'm working at home. And if any of this remote working has taught me anything, it's that most of my colleagues don't have steady/reliable tech or connections, so i'd almost want it used pre-emptively as a default so we can spend the rest of those resources on robustness or other qualities. (I realise of course that at the moment none of them have high-grade Nvidia graphics cards, but I'm talking hypothetically in the far off future).
In short, I want a world where the cost to benefit ratio of things is orders of magnitude larger, because things like this let us spend network/resources on things which matter.
Yes, when I'm calling my parent/grandparent one on one I might want to upgrade the signal, but I don't need to see random colleague's face in all their HD glory, or remote people whom I have no idea who they look or sound like anyway (i believe that's also been one of the findings with deepfakes, that you don't notice the eerieness/falseness as much if it's a reference of a face that you don't have pre-determined knowledge of).
I tried to pay attention to what you and others flagged (parts not moving, facial expressions, shoulders, etc.) But I cannot spot anything out of place.
That said, looking at other comments here, you're definitely not the only one
That way the doctoring isn't apparent, you still benefit from the massive bandwidth savings (which I consider most important anyway), and it appears more believable in the real-world context of variable bitrates.
Petapixel is a blog spam site btw. Why not go to the source that is linked in the post?
A nitpick, perhaps, but isn't that three orders of magnitude?
We've already seen people use outlandish backgrounds in calls, now it's going to be possible to design similar outlandish views, but actually be this new invention in real time. There's been a lot of discussion centered around deep fakes and its problems, this is essentially deep faking yourself into whatever you want.
Video calls are a very important form of communication at the moment, if this becomes as accepted as background modification, that would open the societal door to a whole range of self presentation that up till now was restricted to in game virtual characters.
I wonder what kind of implications that could have. Would people come to identify themselves strongly with a virtual avatar, perhaps stronger than their real life "avatar"? It is an awesome freedom to have, to remake yourself.
Thanks for that comment. Can't wait for it to happen !
I dunno that I'd call it three orders - it's close at about 830x - but it's definitely not even close to being one order either.
"An order-of-magnitude difference between two values is a factor of 10. For example, the mass of the planet Saturn is 95 times that of Earth, so Saturn is two orders of magnitude more massive than Earth."
https://en.wikipedia.org/wiki/Order_of_magnitude
"a new car is an order of magnitude difference in price compared to a used car" is appropriate even if a new car is 40k and a used car is 5k
Electric cars have two orders of magnitude less energy storage than gasoline cars, but newer ones are only one order of magnitude.
Perhaps the example was a best-case, and the usual improvement is about 10x. (That or 'order of magnitude' has gone the way of 'exponential' in popular use. I don't think I've noticed that elsewhere, though.)
Viola! My deep fake can stand in at meetings now while I code.
https://youtu.be/wB1X4o-MV6o
Today's meeting was an hour-and-a-half spent on bike-shedding the position--and color--of the "logout" button on our product page.
15 minutes were spent debating the resident usability expert who suggested white text on a dark blue background would be more readable for people with low vision. The department manager insisted on retaining pastel blue as it is his favorite color.
45 minutes were spent arguing over whether "logout" or "log out" comprises proper semantics. Our linguistic expert was unfortunately not able to attend as she was sent on a business trip earlier in the week.
The last 30 minutes were focused on team-building exercises as Bob doodled on his tablet and Susan smiled politely at the speaker whilst screaming internally as she had 3 more meetings to attend before her department could move forward.
WFH hasn't lessened the amount of bullshit, but it's become more tolerable. I mute my mic and put on a playlist with elevator music.
Now all I need is an AI standin and this summarizer and we've basically achieved universal basic income.
45 minutes were spent by humans arguing over whether "logout" or "log out", while I created both buttons (GPT-3 can already do that) and A/B tested it.
Real-time silly hats for people I talk to and I'm sold.
They had no idea.
sure is .. if you stream your face at >30-50Mbit/s. For contrast highest bitrate available on Twitch, used for streaming high motion full screen updating twitchy 1080@60 gaming, is ~6-8Mbit/s.
I mean taking away focus on things that doesn't matter in a virtual meeting such as: Where you are sitting - via Virtual Background Your daily hair style status or if you have a nose pimple - Via NVIDIAs AI showcased here. Would be great.
Though replacing yourself with a "digital" avatar I think takes away many of the benefits an actual live meeting provides.
1. the phenomenon of VTubers https://en.m.wikipedia.org/wiki/Virtual_YouTuber
2. in the virtual animal crossing late night show, Animal Talking, the presenter's (Gary Whitta) avatar doesn't really resemble how the presenter looks in real life https://en.m.wikipedia.org/wiki/Animal_Talking_with_Gary_Whi...
3. I watch a lot of interview s with people in VR Chat and it's very interesting how people seem to find it easier(?) to open up while they are embodying a character. https://youtu.be/KZWOXgc7PA4
Being able to experiment with identity in this way is really interesting to me, and I hope it becomes more mainstream with the proliferation of this technology
Humans becoming more and more dependent on virtual face-to-face meetings and also relying on embellishment of their supposed appearance through the screen. It reminds me of how SciFi authors predicted technology, but with a complimentary commentary on human psychology.
Sorry if it isn't directly related to the post, but it is so striking to me.
Big question though - is this just substituting the problem of not having good internet with not having a really fast nVidia graphics card?
In fact, I struggle to see any difference from the original. It totally feels genuine, I'm not sure why you perceive them to be npcs
This is not fundamentally different from other lousy compression algorithms.
The bits that are missing are reconstructed using a bias built in the compression algorithm. Deep learning based algorithms simply have a more realistic bias, so the artifacts that introduced to replace the missing bits are less noticeable.
In his universe, both the interstellar net and combat links between ships are low bandwidth. Hence, video is interpolated between sync frames or recreated from old footage. Vinge calls the resulting video "evocations".
The plot point being that when the bandwidth gets too low, the interpolation AI has to make lots of stuff up, you are not quite sure exactly what was said.
I seem to remember the bandwidth in the book was very tiny, small number of bits per second (?) so the AI was taking the speech and compressing it into something more compressed than text then decompressing it at the other end into something that was more or less the same.
I believe this is huge and would create higher engagement if everybody was acutally looking into the camera instead of to the side or up all the time. Creating a more human an emotional relation with the people you are talking to.
https://techxplore.com/news/2019-06-intel-eye-contact-video-...
It has a very very cool twist to explain the Fermi Paradox and is a really good example of a universe with one modified rule.
> Called “Free View,” this would allow someone who has a separate camera off-screen to seemingly keep eye contact with those on a video call.
Am I the only one who thinks eye contact on video calls feels creepy? I think I would prefer this feature to remove eye contact on video calls rather than add it.
So a system that maintained eye contact continuously would indeed risk looking creepy!
[1] https://www.forbes.com/sites/carolkinseygoman/2014/08/21/fac... [2] https://www.businessinsider.com/heres-how-long-you-should-ho...
https://blog.emojipedia.org/apples-new-animoji/
And how well does that work when you switch to screen sharing?
The bandwidth and requirement to participate correlate.
Knowledge workers are all about the knowledge that they build up over time when working with a particular environment (be it an industry, a system, or a person/group of people). That knowledge is non-deterministically synthesised in the brain based on the experiences of that person, and being non-deterministic, no AI will come to the same conclusions about every item as this particular human would.
In that case, an emulated personality that is meant to make themselves available as your replacement will be an impostor. One that is less of an expert than yourself at best, and at worst one that is misinformed or misled on various issues (which in turn causes other people to be misled or misinformed).
If the goal is to make it seem like your present when you aren’t, I can believe we’re only a few years away from Reynolds‘ Betas — a Markov chain can’t mimic a human well, but it can mimic a human; GPT3 can do better, and while it still isn’t great, the main reason it feels like it might not be enough for public figures is how easy it is to get it to answer as if it were someone else rather than as The Right Honourable Sir Obvious Madeupname, MP for Oxbridge-upon-Wells who is paying for the chatbot to mimic him in particular.
Perhaps the nets they’re using are compressing out facial microexpressions and when we see it, it seems just a little unnatural. Compression artifacts might be preferable because the information they’re missing is more obvious and less artificial. In other words i’d rather be presented with something obviously flawed than something i can’t quite tell what is wrong.
It's effectively a motion-mapped keypoints of the person projected onto a simulated model. I'm assuming the cartoonish avatar was used as an example to partly avoid drawing direct lines to the full implications.
- There's no reason this couldn't extend to voice modelling as well. (much clearer speaking at much lower bandwidth)
- There's no reason this couldn't extend to replacing your sent projection with another image (or person)
- Professional looking suit wearing presentation when you're nude/hungover/unshaven. Hell, why even stop at using your real gender or visage? Imagine a job interview where every candidate, by definition, visually looked the same :)
- There's no reason you couldn't replace other people's avatar with one's of your own choosing as well.
- Why couldn't we model the rest of the environment?
Not there today, but this future is closer than many realise.
This would be interesting as an upgrade to the “name on resume” test.
Could also see a future company policy that runs peoples data through a “sameness” filter before letting them into the company to scrub bias.
Recall the promise of 5G is an order of magnitude increase in speed (among other things like low latency).
If we can get there by reducing bandwidth requirements by an order, that will be great. Wonder if it applies to Netflix...
Not sure if I understood your post correctly, but you're slightly misled here.
Watch the video. This is not a new general purpose video codec, it is basically Deep Fake - taking a still image (key frame) and superimposing detected facial expression/movement on this key frame (leaving out some technical details).
This is an improvement over (not-anymore-)state-of-the-art h264 since transmission of only a few coordinates mapping your facial expression is significantly less data to transmit then a delta of arbitrary video and periodic keyframes (again generously leaving out important technical details of modern video codecs). Trying to reconstruct a moving car/background/etc. from these facial expression key points will lead nowhere.
Even JPEG takes advantage of human perception, throwing away first what we can’t perceive.
spotty mobile connections with a powerful device (iPhone Android etc...) it might make sense.
https://www.youtube.com/watch?v=t4DT3tQqgRM
But after that, I was reminded of the paranoia (or not?) around Zoom and that, for an extreme example, the CCP was mining and generating facial fingerprints and social networks using video calls. It seems like this technology is the same concept except put to a useful purpose.
And they still don't often enough (see recent ExamSoft issues discussed e.g. here: https://news.ycombinator.com/item?id=24641063 )
We don't really understand just how little information is actually in a photo (we add huge amounts of info in our perception).
My guess is that predictive systems are using contrast as a guide to essentially 3D structures which, simply, just cannot be reconstructed from 2D. And therefore, probably struggle more on dark faces which have different contrast properties.
Now, I don’t know much about neural networks (AI), but my understanding is that if you provide a training set representative of the population makeup, (in America at least) it’ll be biased towards white people as it hasn’t “seen” enough black person images. My limited understanding would then make me think one would need equal white person photos as well as black person photos.
Black faces and white faces are "statistically equivalent" in 3D.
The issue is more, in my view, the hubris of calling this system "facial recognition". It isnt: it's pixel pattern color recognition which sometimes coincides with certain facial patterns.
There's a real network effect with things like codecs - unless some significant proportion of calls can use it, it'll remain a cool but obscure experiment.
I hope Nvidia have the foresight to release something that'll run on any hardware, and under a permissive license, but I suspect not.
The idea is out there already (it's basically deep fake tech, right?), and I'm sure it won't be so long before some open source version of it gets released. Nvidia would be wise to get out in front of that and at least have their brand associated with a widely used variant on the theme.
I made no claims about the cost. :)
I guess I'm also talking about the theoretical maximums in ideal conditions as well, which is sort of cheating on my part. There would be times when you can't get the optimal speed if, like you say, the sun is the way.
The server is in the same city as me to avoid excessive latency.
The benefits of being able to develop anywhere (including from a phone in a pinch), and being able to add extra CPU's and RAM at the click of a button outweigh the need for a network connection for my usecases.