I think he's right about this—things are going to get weird and ugly and people aren't prepared for what's coming. Here's my suggestion for safeguarding your sanity in the years ahead: find the people and the blogs you love, embrace RSS, block or avoid everything else, read old books, stop listening to podcasts, revert to email as the primary channel for occasional "social" correspondence, abandon all side projects that require additional screen time, go outside.
I agree with most of your advice, especially "go outside" ... but I don't understand why one should "stop listening to podcasts". Do you mean stop listening to podcasts which discuss current events? Because there are plenty of good podcasts which don't.
Also, I think this is good advice even without the proliferation of deepfakes or GPT spam. I already abandoned social media in favor of small group chats and Real Life long ago, and I know I'm not alone. The only habit I still need to kick is reading too much HN. ;)
No, this is not the way. What's the point in burying your head in the sand?
I don't think the vast majority of people 1) care about this 2) even know what RSS is 3) will stop listening to podcasts 4) will use email as their primary channel for correspondence.
By all means, do this if it's what you, personally, need.
Hmm. Thinking in this context… Wearing a helmet even in a room of friends looks like overkill from anti-surveillance perspective, but could be a good protection from deep fakes: someone could still record your face with a hidden camera, but it will be useless for fakes because no one knows it's you and on the other hand everyone knows you don't walk around bare face.
For me, I don't see the point. But then I have a personality where that's not a product I would subscribe to or want. I have enough of a hard time connecting with real people. Having meaningless conversations as a paid service with a piece of software is gross to me.
No no no. Much better: read & grok all those unread books on my bookshelf, and then have looong conversations with me about their contents (and closely-related topics) at the times of my choosing. And when you quote from one of those books, don't be afraid to try to adopt the author's own voice.
Assuming we can actually get a realistic human sounding voice... putting that problem (which I'm unconvinced is solved yet) entirely aside for now, there are issues in the text generation side that make this harder than it might look...
The problems I see, in increasing order of difficulty are: smalltalk; memory; and dirty talk.
1st. A lot of the language models are very bad at pointlessness smalltalk that fills time, they aren’t very good at coming up with related conversational prompts or diversions to continue talking about without being carefully steered and purported by an extra layer of supporting software. The deliberately oversimplified example is that ChatGPT waits for you to say something, and will never just ask how your day was. This isn’t particularly difficult but it’s a case of it’s going to feel a lot like sophisticated NPC smalltalk in a video game for a while I suspect due to the need to drive it from a second system that monitors the conversation and provides the supporting timers to keep nudging along a conversation the way they do in video games
2nd. The majority of the models have limited memory and while we’re seeing clever tricks pop up all the time now about how to feed things back into the models prompts in order to keep it focused on a topic or to bring things back up later, it’s going to take another layer of software that try’s to identify key information and store that in a secondary system and then somehow contextually identify opportunities to mention it in order for us to have a chat bot that can remember what sort of things a person might be into, or more importantly not into and regardless of how good the underlying language model gets at coming up with more words to day, unless the overall system it’s driven by can remember that customer 1 likes feet, customer 2 hates feet, and customer 3 is indifferent to feet… the ability to steer the conversation towards “sex” will be haphazard at best and likely extremely vanilla in a way that I suspect might make the entire thing not worth it, since from what I understand, odd fetishes are a large component of the phone sex business along with lonely people wanting someone to talk to who won’t judge them.
3rd. Despite the prodigious amount of time that must be spent engaging in the kinds of conversations that can best be summed up as “dirty talk”, encompassing the entire gradient from flirtation to describing fornication… not a lot of it is recorded, in any way at all. Well that is to say it’s probably “recorded for security” by phone sex operating companies, but the overwhelming majority of this conversational data is not not archived in any form. It’s not kept as audio past whatever length of time the company decides to keep things, it’s not getting transcribed and it’s definitely not getting marked up and annotated in a way that helps train machine learning models on it. There’s obvious legal and ethical issues involved in obtaining this data, which depend on what form it will be in, actual call recordings are obviously more private than a computer generated transcription pipeline that assigns anonymous identifiers to the operators all random anonymising identifiers to all the callers for each call, even when someone might call multiple times… to the best of my knowledge none of this has ever been seriously researched or studied and I’d imagine even with the “limits off” the best out current language models could do is likely to be heavy on the poetry for flirting and for the “sex” part of “phone sex” it would likely swing between overly clinical and heavy on allusion like a mills and boon romance novel sex scene… I’m going to see what I can coax from ChatGPT to provide some extra evidence. I’ll edit if I’m quick enough.
Edit -> Some results:
After a fair amount of coaching, which involving building a "FlirtBot" jailbreak based on the DAN jailbreak prompt, and some stilted back and forth including ChatGPT (ChatGPT 3.5 to be specifi...
Once the generation times get to real-time, I dunno what's going to happen. I follow the voice acting community a lot and this is a big existential threat level of worry in that community.
But I also see some positives for the narrative voice field, but at the expense of actual actors.
The latest sequel to a favorite audio book series has the professional narrator pronouncing different character names and town names entirely different than the previous 7 books.
In another book the narrator completely changed the character voices in the sequel compared to the first.
The positives for listeners is eventually we can guarantee that voices are completely consistent between narrations. An editor will soon be able to describe the emotions of a character and nuance how the AI performs certain scenes.
It's really sad but I think the end of human audio storytelling is coming to and end quite rapidly.
What will happen is humans redefine their role in society, and keep iterating literally on their survival.
We’ve been here before evolving what we call each others roles; priest, cleric, “people name Farmer must be farmers”; they’re arbitrary labels (in that “we don’t know why we exist”, not immutable obligations of reality way), and there’s no reason we to have see this as “the end of human storytelling”; software isn’t lobotomizing us, but changing who can reach us. If info distributed via technology cannot be trusted, good; past has shown there’s no reason to trust technology corporations anyway. Make it explicit.
Death of commercial figurative identity is not literal death and language is not the only source of understanding.
The bots will get all the capital as well as humans get laid off.
What will humans have to offer each other, let alone the corporations?
Next year they are coming out into the real world and replacing your manual labor… and they will never freak out and attack an old lady or lose their patience. And they will be perfectly tailored to everyone’s individual preferences and needs
My guess would be for voice actors to copyright their voices. I am not familiar with the voice generation ML but i guess for things like dramatic pauses a voice actor might still be needed. Also I'm guessing it should be fairly straightforward to detect such copyright violations.
I would really like to see it implemented for heavy text driven rpgs. But overall yeah things look bad for voice actors in general
Conceivably, anyone who uses his or her voice for spoken word or singing could, in theory, cryptographically sign each master copy of the recording, and therefore certify that they were present and active in its creation. Of course any post-processing would lose this attestation. So it's practically useless, I suppose. But it would be a first-line of defense in copyright cases, at least, by being able to say "I did indeed say that, and here's the signature." or "You can't prove that I said and signed that."
It's a use case for merkle trees. Since the mere existence of the hash at a reasonably well-attested time is sufficient, you don't need the rest. A handful of central roots of various trees is fine, morally equivalent to announcing things with an ad in the newspaper.
Now that I think about it, it seems to be a great idea - a verifiable record that some digital artifacts have existed at certain times. Of course it wouldn't be able to "prove ownership" or "prove being original" or other stupid ideas tokenbros throw around, that's impossible. But it wouldn't be profitable for the tokenbros and actually all the distributed attribute is not a requirement. We can probably set up a centralised append only DB which will periodically publish it's own hashes to attest that it wasn't tampered. Probably a useful tool for our deep fake future, the only question is how to scale it, who can be trusted and relied upon to input hashes in the system.
This is a thing I've started at pretty hard. Embedding a signal that survives basic postprocessing isn't that hard, but doing it in such a way that it stays well below the noise floor and survives extensive editing is not easy. If anyone is interested I could be convinced to put my work on GitHub, highly doubtful that it will go commercial unless I get a huge raft of free time.
Are you saying that a cryptographic signature could survive postprocessing? Well, it may, but when you post-process a sound clip, that sound changes itself. I mean, imagine playing cutups with a dialogue track; that actor may or may not want to attest that the final result is theirs. That's the whole thing. If you cut out three words of dialogue, it is no longer cryptographically signed.
SAG created a rule after Crispin Glover’s face was used without permission via a prosthetic in Back to the Future 2 to ensure it never happened again.
I could easily see a similar rule being added by an appropriate union to their contracts to prevent AI copies of voices that weren’t approved by the original actor.
You already have the right to your likeness, which includes the things you say in your voice profile (especially when attached to your name, and not said in a manner like that of a parody or an impersonation for laughs).
The threat is that, supposedly, pretty much all of the media companies that need Voice actors have a clause that allows them to use generative voice AI after the VA's death or if they're otherwise incapacitated to the point where they would be physically unable to get the voice actor back in the studio to record more lines. This is not outlandish for voice actors since they probably couldn't care less about their voice being used after they are dead, but it does mean we'll be hearing a lot more of it as more and more decade-long franchises use it to avoid hiring a good-enough impersonation or voice replacement once the old one dies.
I tried ElevenLabs and it is truly amazing. You don't need much audio to train it and the final result can be incredible. I shared some of the snippets with friends and relatives and after the intitial scare (same as John) we agreed that the outcome wouldn't be different than a few years ago...
We had impersonators for decades, so what have stopped political parties to hire an impersonator to create fake audios? The IA in this case is going to make it more accesible, but in political parties you don't want thousands of fake audios (they would lose credibility), you need only one.
Also, photoshopping photos would have a similar effect, and that technique has been available for years as well.
The difference is the scale, speed, and cost. You have to at least find an impersonator who can do a passable impersonation of the target, which may be a tall order. They can't impersonate everyone perfectly, and the really good ones are probably busy and/or expensive. Anyone can use this. Kids could use it.
Is it? The examples given in the OP are remarkably uninspired, specifically mentioning "a recording of Joe Biden forgetting his own name or what year it is, or Kamala Harris claiming to be running an abortion clinic." At that level of course you can already easily find a high-quality impersonator if you wanted to.
What kinds of stuff are kids going to use this for that is dangerous? Fake their parents to call in sick to school?
The point isn't that kids will get up to no good, it's that literally anyone can use it. Cheaply, easily, quickly, effectively. Tell you what, you hire an impersonator to say naughty things as 10 high profile people, and I'll use AI to do the same and we can compare results!
The samples are amazing, but it doesn't support other languages yet? And you have to pay to try clone a voice via sample MP3/4 file. Also, I hope it supports more payment methods later e.g. Paypal, for one time fee.
I put some Shakespeare into the test on their main page and it was surprisingly good at times. Hearing an AI emote "Out, out, brief candle!" makes me feel strange. This has to replace voice actors in gaming at least. You could change the text as needed and perhaps just add some hints to get the emotion you want.
I feel these kinds of tools really need a way to put "emotional clue" besides text. Just like the instructions that you give to the voice actors. I don't think force the writer to put all the subtle emotional clues as explicit dialogue text is a good idea.
What I have read people do is that they prime the text with something obviously emotional before hand, including punctuation, before their desired output.
They then trim the audio they want to remove the prime phrase.
So something along the lines of:
"Remove this audio because it makes me super angry! Very angry! Now I say what I want to keep!"
Just having the AI read a script is often not enough alone without post processing and manipulation, often it is somewhat flat
Imagine an AI voice of Trump that phones lonely old people to thank them for coming to his rally. After 20 minutes of small talk he asks them to donate to his campaign via bitcoin...
I'd imagine an AI could quite patiently walk them through how to set it up and link their bank accounts. Honestly, it could be really helpful for tech support.
I know at least a couple. Doesn't take much sophistication with all the hype the last few years and different apps making it easy. It's essentially as easy as using any other mobile payment tech e.g. mpesa or alipay
Even simpler. Have the voice give them a short onboarding call, but then have it regularly call them again using reverse charging, and then chat with them. Convince them that they've formed a friendship.
I'm sure it's going to get exponentially better, but right now ChatGPT sounds like one very vanilla guy. The other day I asked it to write me a conversation between David Foster Wallace and Ernest Hemingway, and it came out something like (paraphrasing):
HEMINGWAY: I notice you write about political issues and philosophy.
WALLACE: Thank you, I noticed you write about raw experience and manliness.
HEMINGWAY: Well, whatever our differences I feel like we can get along and work together. We will soon be fast friends.
Has ChatGPT actually read any books (apart from snippets on the Internet), or has that been avoided for copyright reasons?
I wonder how long until someone pirates every ebook on Z-Library or SciHub and trains a language model off that. Imagine a language model that's read every book and every scientific paper.
I wonder if that would even generate good outcomes given how much of the data in some disciplines is conflicting, and if you don’t have a good way to get it to ignore bad subsets of that data it could be more likely to hallucinate convincing scientific sounding explanations based on a preponderance of bad poorly researched evidence.
There’s a lot of papers that discredit older papers or work or entire subfields of study and the current large language models would just all all of it and go “these are words” with no analytical reasoning applied.
These are predictive mechanisms not analytical ones and until we get a better handle on how we can make them “stop and think”… I’m just not sure how much benefit each larger dataset will add beyond “sounds more human” while it’s core problems of “still hallucinates” and “can’t do basic reasoning” remain.
Also one question I’ve got on this front is how much duplicate data is processed out of the input sets? Using LibGen as an example there’s a lot of books that get reprinted and uploaded multiple times in different formats… does having these remain in the data provide a “valuable bias” or is it something that needs removing? Do the people building large language models pre-process their data to de-duplicate out these kinds of things from their current data sources?
I asked it to make a certain stand up bit in the style of various comedians. The Seinfeld one was reasonably good but the Chris Rock one was off because it wouldn’t curse.
It's kind of amazing we're not seeing more of this already. It's a trivial application of the tech. We've already achieved a culture where the more batty something is the more likely people are to believe it, and now we have an infinite, personalized quackery generator.
Without a massive cultural revolution putting the brakes on this, which I can't imagine coming from anywhere, I fear we're barreling into a future where nothing is true anymore, a hall of mirrors where the capacity to fake anything at all far surpasses anyone's ability to make sense of it.
MacLuhan and others have already talked about this, from way back in the radio and TV generation.
I don't say this to mean, "oh, and they were wrong because everything turned out OK" If anything, we have had generations that grew up with this dysfunction in a normalized way, and it took another advanced in tech to really see it again.
(To take it further back into history, I think I now get what Cervantes was trying to say with the image of Don Quxiote tilting at windmills. It seemed so absurd, amusing, and quaint when I read it as a teen).
Would it really be wise to run a criminal operation while piping all your requests through Microsoft servers, considering the history of the company collaborating with law enforcement?
We can just make the AI phishing bots talk to a Lenny 2.0 trained to tie them up in endless conversations. AI interacting with AI will be the next economic boom fomenting a wave of new wealth creation.
"Hey mom, it's X. Sorry to leave this voice mail, I got in a bit of a car accident and lost my phone. I need to get back home, could you send $$$$ to ####?"
This wouldn't work for me. I always pretend to like something and then change my mind the next day to piss people off.
Maybe just a call to ensure it's legit. Vocal deepfakes will never mimic voices and ways of talking in real time, it will always sound "off". It doesn't make it any less dystopian, though.
> It’s all fun and games in these demos, but this is inevitably going to be put to use by ratfuckers to create fake scandals in political campaigns...And it feels inevitable that a Roger Stone or Steve Bannon type will use this technology to commission, say, a recording of Joe Biden forgetting his own name...
The road to hell is paved with good intentions, and all politicians are going to use this for their political campaigns, not just one specific side. Both.
Let's not veer off the wider point and believe that only one side will use it for bad things. All politicians are liars, and no matter what side they are on, if it benefits their agenda to influence the electorate to gain power, they will use it; even if it is used for spreading lies or false and misleading claims.
Mark Elias. And the dozens of people who perpetrated a massive Russia collusion hoax on America so successfully that some people still don’t realize they were fooled into oblivion.
Sure. The BLM fraudsters Alicia Garza, Patrisse Cullors, and Opal Tometi whose grifting campaigns brought in $180M for the organization only for them to spend it on themselves. More than what Roger Stone or Steve Bannon's worth combined.
Again. Let us not pretend that only one side can use this for their next so called 'donations' or 'fundraising campaign'. Both of them will, to influence the electorate of their choice back into power.
When this topic has come up with family and friends, folks often say that they aren't worried (yet) because while a human can be fooled, they can't yet fool readily available forensic tools, and perhaps can't ever.
I can't speak to the veracity of that claim, but as the post points out, the past several years has shown us that it doesn't matter, not in the least.
The author goes on to say how it feels inevitable that we'll see a Bannon or Stone type use this technology to create fake scandals.
I'm more worried about the grass roots efforts. Crowdsourced conspiracies like QAnon. Now they'll have more capable tools to radicalize people.
On the flip side, if these deepfakes are so simple to generate, they will surely bombard us constantly throughout the day - radio, TV commercials, youtube videos, podcasts.
And people quickly become desensitized to that kind of thing. It could be the case that, after some initial "ramping up" period, these deepfakes are so cheap and abundant that no one falls for them.
When was the last time you got a call from a number you didn't recognize, and you actually picked it up? If you're like me, probably not in a long time, because you're aware it's almost certainly some scam/robo call.
It is scary for sure. We were heading towards a brave new world even before chatGPT, where what's real was no longer to be taken for granted and you couldn't trust your eyes and ears. ChatGPT has just hastened things. I'm sure a new market will emerge for authentication services that verify the speaker or the person in the video similar to how we had twitter verified checkmark.
On the flip side I can't wait for someone to build a product where I record a few conversations with my parents while they are alive, and then when they're gone, through chatGPT + vocalfakes, I can have a parent forever. Sure, I'll know it's not the real thing, but when you really miss them, it certainly can make the pain a little less.
Along this line of thinking, I try to capture a 20 minute video of random conversation during each family vacation. Just set prop up the camera and record.
When we take trips, I like to record a retrospective. Just things like what did you like/dislike/find interesting. They feel awkward but then they don’t.
Everyone's worried about people using this to generate things that people haven't said (The "grab em" comment by Trump) but inversely to this is that you can claim any REAL content about someone was AI generated by someone else.
Was anyone else not particularly impressed by this? I haven't listened to much Steve Jobs stuff but it sounded just stilted enough for me to think something was up. I'm not sure if it was because I came into it with a skeptical mindset because of the article and context around it. It also may be do to the fact that I have been using screen reading software for about 30 years. The only people who I've heard more then synthetic speech may be my close family and I'm not sure about that. Is there anywhere that offers a test where you have to determine what is generated and what is not with random clips lacking background info?
The audio directly in the post was for sure, but did you listen to the audio in the linked twitter thread? The only thing that gave me pause was the gpt text itself not the voice audio.
I had not listened to the audio in the twitter thread. It's a lot better and the only issue I notice is the pauses at the end of sentences don't seem quite right. I don't know if I'd notice the pauses though if I were not going into this with a suspicious mindset. After listening to the twitter thread I'm not convinced I could detect the difference if it was two random clips one generated and one not.
The Twitter audio had a distinct "key note address" cadence to me rather than being conversational. Of course probably most training voice came from such addresses.
One way: Everyone has a code-word which they only share with humans, over "private" channels like direct voice calls. For example, I'm "pineapple", and you're "orange", when we're connected on a call, we share our code-words in order to "authenticate" the call.
Another way: have a passphrase with a group of people that you share with each other, that is hard to guess but memorable. "The sky was green this morning" for example, that everyone in your family knows is the "human passphrase".
Another way: ask personal questions only the actual human being would know, like inside jokes or specific details about their life.
Another way: verify through some out-of-band method (like text message) in order to confirm their identity.
Another way: start everything with a video call where they have to pass their hand in front of their face before switching to audio only
None of the methods are perfect, all come with pro and cons.
But writing all of this made me feel really dystopian for some reason...
Someone should make a family authentication app. It would generate a TOTP secret and associated QR code that could be shared with other family members. (This app does not have to be a mobile app. It could be a desktop app, or even a web app that runs locally in the browser).
Have every family member scan that code for use with the TOTP authenticator of their choice on their phone and/or tablet and/or watch.
When you are talking to someone who purports to be a family member you ask them for the family TOTP code and check to see if they give the same code that your TOTP authenticator on your phone or watch is showing.
Would require a lot of client support, which probably means it's never going to happen, but you could definitely sign the audio stream and have your player display the signature information to match it to a certificate via some https-like root store program where you can verify the audio is signed by the speaker.
This is one of those things where Balaji is ahead of the curve - the way to guarantee metadata (e.g. who the speaker is) is to do it cryptographically on-chain.
the voice data signature can be removed, unless you mean something like an audio watermark
> What via hash
audio remuxing will destroy the hash, especially since it's already done by pretty much every social media company. Even with some sort of lossy audio hash, just put some clapping or cheering behind the voice and you've probably created a new audio hash.
> When via timestamp
file timestamps can change
This solves nothing unless your user either doesn't care about sharing the original file, or is malicious but too dumb to ffmpeg -i original.ogg out.mp3.
There's an original, canonical version of the audio, and the hash of that version goes on chain. That establishes who posted it originally, when, and the exact version. Then any subsequent (potentially doctored) version can be compared with the canonical original. The on-chain timestamp and hash can't be changed after the fact, in the same way that past bitcoin blocks can't be changed (without creating a fork).
It could at least address some kinds of misleading editing. In the political use case, maybe the candidate posts all their event audio and records the hashes on chain. Then they can't change the content of any of it without getting caught. And if someone else posts an edited version, the edit will have a later timestamp, and the candidate can point to their original earlier version and prove that it's the original. That lets everyone else determine which version is the original and which version is the edit.
I've used a tool to make myself sound like a girl, it is freakily realistic and uncanny hearing what it produces. Chinese love scam rings are going to be all over this in a year, coupled with image/video generation
I looked into this a while ago when I was thinking of DMing a TTRPG during the pandemic lockdowns thinking I might be able to use the tech to make having to play the game online less of a negative and more of a fun different experience… and I was extremely disappointed by the majority of them and just generally disappointed by all of the ones I tried. They all had massive shortcomings of one kind or another, and even the most advanced ones where I would have to pre-record audio and upload that to their online charge by the minute of processed audio… they had a noticeable, irritating dullness, like it was trying to remove expressiveness in order to better conform to to their target waveforms and honestly… it sounded less alive than if I just used a text to speech engine with a female voice like the one that powers Siri.
I might be a little biased since I have used a lot of text to speech tools over the years (I routinely listen to hours of TTS generated audio per week to read technical books and research papers that aren’t in audiobook format) and consequently developed a bit of an “ear” for all the ways various TTS engines sound robotic, mangle pronunciation, and generally fail to read how a human would.
I’d be very curious which tool or service you used that sounded “freakily realistic”, care to share?
I’ve been looking for a good one to test out if running the TTS audio through it sounds any more or less robotic, since their robotic prosody gets deeply tiring with particular kinds of text after a while.
This feels so over-the-top. "A recording of Joe Biden forgetting his own name or what year it is, or Kamala Harris claiming to be running an abortion clinic?" Give me a break, you could already have done this with a high-quality voice impersonator -- cost is no real concern at that level anyway.
I think this is only really a risk at the low end -- people scamming others with fake references from quasi-celebrities. Not great, but overall feels pretty minor of a concern. We already allow the carriers to scam everyone in the US by allowing anyone to call your cell phone and try to tell you you're behind on your insurance, or that your computer has a virus. There's plenty of scams out there, if we really thought this was a problem, we'd care more about the existing ones.
I've already been seeing issues with AI art being used to fake Trump's arrest, people are genuinely confused and it even took me a second to realise I was being had
121 comments
[ 4.6 ms ] story [ 147 ms ] threadThis is the way.
Also, I think this is good advice even without the proliferation of deepfakes or GPT spam. I already abandoned social media in favor of small group chats and Real Life long ago, and I know I'm not alone. The only habit I still need to kick is reading too much HN. ;)
I don't think the vast majority of people 1) care about this 2) even know what RSS is 3) will stop listening to podcasts 4) will use email as their primary channel for correspondence.
By all means, do this if it's what you, personally, need.
Hmm. Thinking in this context… Wearing a helmet even in a room of friends looks like overkill from anti-surveillance perspective, but could be a good protection from deep fakes: someone could still record your face with a hidden camera, but it will be useless for fakes because no one knows it's you and on the other hand everyone knows you don't walk around bare face.
The problems I see, in increasing order of difficulty are: smalltalk; memory; and dirty talk.
1st. A lot of the language models are very bad at pointlessness smalltalk that fills time, they aren’t very good at coming up with related conversational prompts or diversions to continue talking about without being carefully steered and purported by an extra layer of supporting software. The deliberately oversimplified example is that ChatGPT waits for you to say something, and will never just ask how your day was. This isn’t particularly difficult but it’s a case of it’s going to feel a lot like sophisticated NPC smalltalk in a video game for a while I suspect due to the need to drive it from a second system that monitors the conversation and provides the supporting timers to keep nudging along a conversation the way they do in video games
2nd. The majority of the models have limited memory and while we’re seeing clever tricks pop up all the time now about how to feed things back into the models prompts in order to keep it focused on a topic or to bring things back up later, it’s going to take another layer of software that try’s to identify key information and store that in a secondary system and then somehow contextually identify opportunities to mention it in order for us to have a chat bot that can remember what sort of things a person might be into, or more importantly not into and regardless of how good the underlying language model gets at coming up with more words to day, unless the overall system it’s driven by can remember that customer 1 likes feet, customer 2 hates feet, and customer 3 is indifferent to feet… the ability to steer the conversation towards “sex” will be haphazard at best and likely extremely vanilla in a way that I suspect might make the entire thing not worth it, since from what I understand, odd fetishes are a large component of the phone sex business along with lonely people wanting someone to talk to who won’t judge them.
3rd. Despite the prodigious amount of time that must be spent engaging in the kinds of conversations that can best be summed up as “dirty talk”, encompassing the entire gradient from flirtation to describing fornication… not a lot of it is recorded, in any way at all. Well that is to say it’s probably “recorded for security” by phone sex operating companies, but the overwhelming majority of this conversational data is not not archived in any form. It’s not kept as audio past whatever length of time the company decides to keep things, it’s not getting transcribed and it’s definitely not getting marked up and annotated in a way that helps train machine learning models on it. There’s obvious legal and ethical issues involved in obtaining this data, which depend on what form it will be in, actual call recordings are obviously more private than a computer generated transcription pipeline that assigns anonymous identifiers to the operators all random anonymising identifiers to all the callers for each call, even when someone might call multiple times… to the best of my knowledge none of this has ever been seriously researched or studied and I’d imagine even with the “limits off” the best out current language models could do is likely to be heavy on the poetry for flirting and for the “sex” part of “phone sex” it would likely swing between overly clinical and heavy on allusion like a mills and boon romance novel sex scene… I’m going to see what I can coax from ChatGPT to provide some extra evidence. I’ll edit if I’m quick enough.
Edit -> Some results:
After a fair amount of coaching, which involving building a "FlirtBot" jailbreak based on the DAN jailbreak prompt, and some stilted back and forth including ChatGPT (ChatGPT 3.5 to be specifi...
But I also see some positives for the narrative voice field, but at the expense of actual actors.
The latest sequel to a favorite audio book series has the professional narrator pronouncing different character names and town names entirely different than the previous 7 books.
In another book the narrator completely changed the character voices in the sequel compared to the first.
The positives for listeners is eventually we can guarantee that voices are completely consistent between narrations. An editor will soon be able to describe the emotions of a character and nuance how the AI performs certain scenes.
It's really sad but I think the end of human audio storytelling is coming to and end quite rapidly.
We’ve been here before evolving what we call each others roles; priest, cleric, “people name Farmer must be farmers”; they’re arbitrary labels (in that “we don’t know why we exist”, not immutable obligations of reality way), and there’s no reason we to have see this as “the end of human storytelling”; software isn’t lobotomizing us, but changing who can reach us. If info distributed via technology cannot be trusted, good; past has shown there’s no reason to trust technology corporations anyway. Make it explicit.
Death of commercial figurative identity is not literal death and language is not the only source of understanding.
The bots will get all the capital as well as humans get laid off.
What will humans have to offer each other, let alone the corporations?
Next year they are coming out into the real world and replacing your manual labor… and they will never freak out and attack an old lady or lose their patience. And they will be perfectly tailored to everyone’s individual preferences and needs
https://www.axios.com/2023/03/17/robots-humanoid-figure-tesl...
I would really like to see it implemented for heavy text driven rpgs. But overall yeah things look bad for voice actors in general
SAG created a rule after Crispin Glover’s face was used without permission via a prosthetic in Back to the Future 2 to ensure it never happened again.
I could easily see a similar rule being added by an appropriate union to their contracts to prevent AI copies of voices that weren’t approved by the original actor.
The threat is that, supposedly, pretty much all of the media companies that need Voice actors have a clause that allows them to use generative voice AI after the VA's death or if they're otherwise incapacitated to the point where they would be physically unable to get the voice actor back in the studio to record more lines. This is not outlandish for voice actors since they probably couldn't care less about their voice being used after they are dead, but it does mean we'll be hearing a lot more of it as more and more decade-long franchises use it to avoid hiring a good-enough impersonation or voice replacement once the old one dies.
Relevant speculative fiction that tops the current /wsg/ thread: https://www.youtube.com/watch?v=-gGLvg0n-uY
We had impersonators for decades, so what have stopped political parties to hire an impersonator to create fake audios? The IA in this case is going to make it more accesible, but in political parties you don't want thousands of fake audios (they would lose credibility), you need only one.
Also, photoshopping photos would have a similar effect, and that technique has been available for years as well.
What kinds of stuff are kids going to use this for that is dangerous? Fake their parents to call in sick to school?
My impression about these voice generators was they're only good for Youtube tutorials and such. It was 2 years ago tho, I wonder how things changed.
So something along the lines of:
"Remove this audio because it makes me super angry! Very angry! Now I say what I want to keep!"
Just having the AI read a script is often not enough alone without post processing and manipulation, often it is somewhat flat
I wonder if ChatGPT combined with a voice generator can mount successful phishing campaigns.
It looks like Cyberpunk future we never asked is already here.
Color me skeptical.
[0] https://www.atmmarketplace.com/news/elderly-michigan-couple-...
HEMINGWAY: I notice you write about political issues and philosophy.
WALLACE: Thank you, I noticed you write about raw experience and manliness.
HEMINGWAY: Well, whatever our differences I feel like we can get along and work together. We will soon be fast friends.
I wonder how long until someone pirates every ebook on Z-Library or SciHub and trains a language model off that. Imagine a language model that's read every book and every scientific paper.
There’s a lot of papers that discredit older papers or work or entire subfields of study and the current large language models would just all all of it and go “these are words” with no analytical reasoning applied.
These are predictive mechanisms not analytical ones and until we get a better handle on how we can make them “stop and think”… I’m just not sure how much benefit each larger dataset will add beyond “sounds more human” while it’s core problems of “still hallucinates” and “can’t do basic reasoning” remain.
Also one question I’ve got on this front is how much duplicate data is processed out of the input sets? Using LibGen as an example there’s a lot of books that get reprinted and uploaded multiple times in different formats… does having these remain in the data provide a “valuable bias” or is it something that needs removing? Do the people building large language models pre-process their data to de-duplicate out these kinds of things from their current data sources?
Without a massive cultural revolution putting the brakes on this, which I can't imagine coming from anywhere, I fear we're barreling into a future where nothing is true anymore, a hall of mirrors where the capacity to fake anything at all far surpasses anyone's ability to make sense of it.
I don't say this to mean, "oh, and they were wrong because everything turned out OK" If anything, we have had generations that grew up with this dysfunction in a normalized way, and it took another advanced in tech to really see it again.
(To take it further back into history, I think I now get what Cervantes was trying to say with the image of Don Quxiote tilting at windmills. It seemed so absurd, amusing, and quaint when I read it as a teen).
What we are growing now is a system that can give each and every one of us our own reality, and if it succeeds, what of society?
Years and years of bad software practices, corporate cost cutting measures resulted in a planet scale cybersecurity mess.
Current hacking scene is more similar to 80s than early 2000s, it is extremely hard to catch advanced actors.
They did this in one of the Harry Potter books (the sixth one?) since it is easy for them to impersonate other people.
If you can’t tell me my favorite flavor of jam then you aren’t getting my money.
The road to hell is paved with good intentions, and all politicians are going to use this for their political campaigns, not just one specific side. Both.
Let's not veer off the wider point and believe that only one side will use it for bad things. All politicians are liars, and no matter what side they are on, if it benefits their agenda to influence the electorate to gain power, they will use it; even if it is used for spreading lies or false and misleading claims.
Again. Let us not pretend that only one side can use this for their next so called 'donations' or 'fundraising campaign'. Both of them will, to influence the electorate of their choice back into power.
What are the "types" that would do something like this?
https://www.bbc.com/news/world-us-canada-59168626
https://www.theguardian.com/us-news/2022/oct/11/russian-anal...
https://www.nytimes.com/2022/10/18/us/politics/igor-danchenk...
https://www.npr.org/2021/11/12/1055030223/the-fbi-arrests-a-...
I can't speak to the veracity of that claim, but as the post points out, the past several years has shown us that it doesn't matter, not in the least.
The author goes on to say how it feels inevitable that we'll see a Bannon or Stone type use this technology to create fake scandals.
I'm more worried about the grass roots efforts. Crowdsourced conspiracies like QAnon. Now they'll have more capable tools to radicalize people.
[Clip of Abe Lincoln in RayBans saying "Don't believe everything you see and hear on the Internet" goes here]
Even the stuff that wasn't deepfaked was mostly bollocks anyway.
Generated images and audio clips are still pretty easy to detect, we're not there yet, but close.
What I worry about is not the obvious usage and risks, but the second and third order effects, those we can't predict.
I will be interesting.
And people quickly become desensitized to that kind of thing. It could be the case that, after some initial "ramping up" period, these deepfakes are so cheap and abundant that no one falls for them.
When was the last time you got a call from a number you didn't recognize, and you actually picked it up? If you're like me, probably not in a long time, because you're aware it's almost certainly some scam/robo call.
On the flip side I can't wait for someone to build a product where I record a few conversations with my parents while they are alive, and then when they're gone, through chatGPT + vocalfakes, I can have a parent forever. Sure, I'll know it's not the real thing, but when you really miss them, it certainly can make the pain a little less.
When we take trips, I like to record a retrospective. Just things like what did you like/dislike/find interesting. They feel awkward but then they don’t.
https://mastodon.social/@jamesthomson/110062947060928918
I had no idea things had gone this far.
Another way: have a passphrase with a group of people that you share with each other, that is hard to guess but memorable. "The sky was green this morning" for example, that everyone in your family knows is the "human passphrase".
Another way: ask personal questions only the actual human being would know, like inside jokes or specific details about their life.
Another way: verify through some out-of-band method (like text message) in order to confirm their identity.
Another way: start everything with a video call where they have to pass their hand in front of their face before switching to audio only
None of the methods are perfect, all come with pro and cons.
But writing all of this made me feel really dystopian for some reason...
and also they have to turn to profile.
Have every family member scan that code for use with the TOTP authenticator of their choice on their phone and/or tablet and/or watch.
When you are talking to someone who purports to be a family member you ask them for the family TOTP code and check to see if they give the same code that your TOTP authenticator on your phone or watch is showing.
i dont know how many videos it will take of trump and biden shit talking eachother on xbox live but it looks like several on youtube are near 1m views
https://www.youtube.com/watch?v=AmUC4m6w1wo
[0] https://youtu.be/WhsSzIS84ks
https://twitter.com/balajis/status/1583495595737481217
the voice data signature can be removed, unless you mean something like an audio watermark
> What via hash
audio remuxing will destroy the hash, especially since it's already done by pretty much every social media company. Even with some sort of lossy audio hash, just put some clapping or cheering behind the voice and you've probably created a new audio hash.
> When via timestamp
file timestamps can change
This solves nothing unless your user either doesn't care about sharing the original file, or is malicious but too dumb to ffmpeg -i original.ogg out.mp3.
It could at least address some kinds of misleading editing. In the political use case, maybe the candidate posts all their event audio and records the hashes on chain. Then they can't change the content of any of it without getting caught. And if someone else posts an edited version, the edit will have a later timestamp, and the candidate can point to their original earlier version and prove that it's the original. That lets everyone else determine which version is the original and which version is the edit.
I might be a little biased since I have used a lot of text to speech tools over the years (I routinely listen to hours of TTS generated audio per week to read technical books and research papers that aren’t in audiobook format) and consequently developed a bit of an “ear” for all the ways various TTS engines sound robotic, mangle pronunciation, and generally fail to read how a human would.
I’d be very curious which tool or service you used that sounded “freakily realistic”, care to share?
I’ve been looking for a good one to test out if running the TTS audio through it sounds any more or less robotic, since their robotic prosody gets deeply tiring with particular kinds of text after a while.
Are you able to share the name of the voice changer?
I think this is only really a risk at the low end -- people scamming others with fake references from quasi-celebrities. Not great, but overall feels pretty minor of a concern. We already allow the carriers to scam everyone in the US by allowing anyone to call your cell phone and try to tell you you're behind on your insurance, or that your computer has a virus. There's plenty of scams out there, if we really thought this was a problem, we'd care more about the existing ones.