Has anyone considered decompiling eloquence? With something like ghidra or ida pro? Mario 64 was turned back into high level language source code this way.
This surprises me: "These modern systems are developed to sound human, natural, and conversational. Unfortunately this seems to come at the expense of accuracy. In my testing, both models had a tendency to skip words, read numbers incorrectly, chop off short utterances, and ignore prosody hints from text punctuation. "
So what's the way forward for blind screen reader users? Sadly, I don't know.
Modern text to speech research has little overlap with our requirements. Using Eloquence [32-bit voice last compiled in 2003], the system that many blind people find best, is becoming increasingly untenable. ESpeak uses an odd architecture originally designed for computers in 1995, and has few maintainers. Blastbay Studios [...] is a closed-source product with a single maintainer, that also suffers from a lack of pronunciation accuracy.
In an ideal world, someone would re-implement Eloquence as a set of open source libraries. However, doing so would require expertise in linguistics, digital signal processing, and audiology, as well as excellent programming abilities. My suspicion is that modernizing the text to speech stack that is preferred by blind power-users is an effort that would require several million dollars of funding at minimum.
Instead, we'll probably wind up having to settle for text to speech voices that are "good enough", while being nowhere near as fast and efficient [800 to 900 words per minute] as what we have currently.
What use is human sounding TTS when your desktop cannot read the contents of windows?
As someone with progressive retinal tearing who's used the linux desktop for 20 years I'm terrified. The forcing of the various incompatible waylands by the big linux corps has meant the end of support for screen readers. The only wayland compositor that supports screen readers in linux is GNOME's mutter and they literally only added that support last year (after 15 years of waylands) and instead of supporting standard at-spi and existing protocols that Orca and the like use GNOME decided to come up with two new in-house GNOME proprietary protocols (which themselves don't send the full window tree or anything on request but instead push only info about single windows, etc, etc) for doing it. No other wayland compositor supports screen readers. And without any standardization no developers will ever support screenreaders on waylands. Basically only GNOME's userspace will sort of support it. There's no hope for non-X11 based screen readers and all the megacorps are say they're dropping X11 support.
The only options I have are to use and maintain old X11 linux distros myself. But eventually things like CA TLS and browsers just won't be feasible for me to backport and compile myself. Eventually I'm going to have to switch to using Windows. It's a sad, sad state of things.
And regarding AI based text to speech: almost all of it kind of sucks for screen readers. Particularly the random garbled ai-noises that happen between and at the end of utterances, inaccurate readings, etc in many models. Not to mention requiring the use of a GPU and lots of system resources. The old Festival 1.96 Nitech HTS voices on (core2duo) CPU from the early 2000 are incomparibly faster, more accurate, and sound decent enough to understand.
It's not just screen reader users. I use TTS to listen to text content and the AI TTS voices I've tried have the issues with skipping words or generating garbled output in sections.
I don't know if this is a data/transcription issue, an issue with noisy audio, or what.
I have been working on playing around with over 10 stt systems in last 25 days and its really weird to read this article as my experience is the opposite. Stt models are amazing today. They are stupid fast, sound great and very simple to implement as huggingface spaces code is readily available for any model. Whats funny is that the model he was talking about "supertonic" was exactly the model I would have recommended if people wanted to see how amazing the tech has become. The model is tiny, runs 55x real time on any potato and sounds amazing. Also I think he is implementing his models wrong. As he mentions that some models don't have streaming and you have to wait for the whole chunk to be processed. But that's not a limit in any meaningful way as you can define the chunk. You can simply make the first n characters within the first sentence be the chunk and process that first and play that immediately while the rest of the text is being processed. ttfs and ttfa on all modern day models is well below 0.5 and for supertonic it was 0.05 with my tests.....
This almost perfectly encapsulates the problems that create friction for new technology. People want/expect the new technology to be an upgraded version of the old technology.
"AI is going to make screen readers amazing!"
No, that is not what AI is going to do. That is the exact kind of missing the forest for the trees that comes with new tech.
AI will be used to act as a sighted person sitting next to the blind person, who the blind person is conversing with (at whatever speed they wish) to interpret and do stuff on the screen. It's a total misapplication of AI to think the goal is to leverage it to make screen readers better.
They can have sighted servant who is gleefully collaborating with them to use their computer. You don't need 900 words per minute read to you so you can build a full mental model of every webpage. You can just say "Lets go on amazon and look for paper towels", "Lets check the top stories on HN"
AI in this sense means using Machine Learning (ML)/Neural Networks (NN) to convert the text (or phonemes) to audio.
There are effectively two approaches to voice synthesis: time-domain and pitch-domain.
In time-domain synthesis you care concatenating short waveforms together. These are variations of Overlap and Add: OLA [1], PSOLA [2], MBROLA [3], etc.
In pitch-domain synthesis, the analysis and synthesis happens in the pitch domain through the Fast Fourier Transform (visualized as a spectrogram [4]), often adjusted to the Mel scale [5] to better highlight the pitches and overtones. The TTS synthesizer is then generating these pitches and converting them back to the time domain.
The basic idea is to extract the formants (pitch bands for the fundamental frequency and overtones) and have models for these. Some techniques include:
Does having it sound "natural" even matter for high-speed reading? I assumed it would be a hindrance at higher speeds because natural variation and randomness in a voice makes it harder to scan the voice (similar to how reading something handwritten tends to be harder than something that has been typeset). At least that's how I always feel whenever I listen to audiobooks that use "natural" voices - I always switch to the more robotic sounding ones because, in my experience, it's easier to scan once at 2x and beyond.
My takeaway from the article is that accuracy of pronunciation, tweakability, and "time to first utterance" are what matter most.
Natural-sounding AI is like fancy cursive font for writing code, it slows things down. The right tool fits the job, and the job here is information retrieval.
I've been using a screen reader Chrome extension for 15 years using the Alex voice on MacOS. Some people find it robotic but I could not replace it yet. I speed it up to 1.4x. When I tried Eloquence voice now it sounded even more robotic, but I can relate to that.
Funny I've actually been digging into this problem recently. I have a webaudio reimplementation of Klatt 1980 driven by cmudict. It still sounds pretty ass, but it's very early days. This weekend I intend to go deep dive on the Delta rule system that powers Eloquence. There're so many interesting papers from the late 90s early 2000s I bet we could get something pretty remarkable that sounds even better than Eloquence and is incredibly fast and runs anywhere.
18 comments
[ 2.3 ms ] story [ 43.6 ms ] threadSo what's the way forward for blind screen reader users? Sadly, I don't know.
Modern text to speech research has little overlap with our requirements. Using Eloquence [32-bit voice last compiled in 2003], the system that many blind people find best, is becoming increasingly untenable. ESpeak uses an odd architecture originally designed for computers in 1995, and has few maintainers. Blastbay Studios [...] is a closed-source product with a single maintainer, that also suffers from a lack of pronunciation accuracy.
In an ideal world, someone would re-implement Eloquence as a set of open source libraries. However, doing so would require expertise in linguistics, digital signal processing, and audiology, as well as excellent programming abilities. My suspicion is that modernizing the text to speech stack that is preferred by blind power-users is an effort that would require several million dollars of funding at minimum.
Instead, we'll probably wind up having to settle for text to speech voices that are "good enough", while being nowhere near as fast and efficient [800 to 900 words per minute] as what we have currently.
I feel like there’s a lot of backstory I’m missing.
As someone with progressive retinal tearing who's used the linux desktop for 20 years I'm terrified. The forcing of the various incompatible waylands by the big linux corps has meant the end of support for screen readers. The only wayland compositor that supports screen readers in linux is GNOME's mutter and they literally only added that support last year (after 15 years of waylands) and instead of supporting standard at-spi and existing protocols that Orca and the like use GNOME decided to come up with two new in-house GNOME proprietary protocols (which themselves don't send the full window tree or anything on request but instead push only info about single windows, etc, etc) for doing it. No other wayland compositor supports screen readers. And without any standardization no developers will ever support screenreaders on waylands. Basically only GNOME's userspace will sort of support it. There's no hope for non-X11 based screen readers and all the megacorps are say they're dropping X11 support.
The only options I have are to use and maintain old X11 linux distros myself. But eventually things like CA TLS and browsers just won't be feasible for me to backport and compile myself. Eventually I'm going to have to switch to using Windows. It's a sad, sad state of things.
And regarding AI based text to speech: almost all of it kind of sucks for screen readers. Particularly the random garbled ai-noises that happen between and at the end of utterances, inaccurate readings, etc in many models. Not to mention requiring the use of a GPU and lots of system resources. The old Festival 1.96 Nitech HTS voices on (core2duo) CPU from the early 2000 are incomparibly faster, more accurate, and sound decent enough to understand.
I don't know if this is a data/transcription issue, an issue with noisy audio, or what.
Is supertonic the best sounding model, or is there a different one you'd recommend that doesn't perform as well but sounds even better?
"AI is going to make screen readers amazing!"
No, that is not what AI is going to do. That is the exact kind of missing the forest for the trees that comes with new tech.
AI will be used to act as a sighted person sitting next to the blind person, who the blind person is conversing with (at whatever speed they wish) to interpret and do stuff on the screen. It's a total misapplication of AI to think the goal is to leverage it to make screen readers better.
They can have sighted servant who is gleefully collaborating with them to use their computer. You don't need 900 words per minute read to you so you can build a full mental model of every webpage. You can just say "Lets go on amazon and look for paper towels", "Lets check the top stories on HN"
There are effectively two approaches to voice synthesis: time-domain and pitch-domain.
In time-domain synthesis you care concatenating short waveforms together. These are variations of Overlap and Add: OLA [1], PSOLA [2], MBROLA [3], etc.
In pitch-domain synthesis, the analysis and synthesis happens in the pitch domain through the Fast Fourier Transform (visualized as a spectrogram [4]), often adjusted to the Mel scale [5] to better highlight the pitches and overtones. The TTS synthesizer is then generating these pitches and converting them back to the time domain.
The basic idea is to extract the formants (pitch bands for the fundamental frequency and overtones) and have models for these. Some techniques include:
1. Klatt formant synthesis [6]
2. Linear Predictive Coding (LPC) [7]
3. Hidden Markov Model (HMM) [8]
4. WaveGrad NN/ML [9]
[1] https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method
[2] https://en.wikipedia.org/wiki/PSOLA -- Pitch-synchronous Overlap and Add
[3] https://en.wikipedia.org/wiki/MBROLA -- Multi-Band Resynthesis Overlap and Add
[4] https://en.wikipedia.org/wiki/Spectrogram
[5] https://en.wikipedia.org/wiki/Mel_scale
[6] https://en.wikipedia.org/wiki/Dennis_H._Klatt
[7] https://en.wikipedia.org/wiki/Linear_predictive_coding
[8] https://www.cs.cmu.edu/~awb/papers/ssw6/ssw6_294.pdf
[9] https://arxiv.org/abs/2009.00713 -- WaveGrad: Estimating Gradients for Waveform Generation
My takeaway from the article is that accuracy of pronunciation, tweakability, and "time to first utterance" are what matter most.
There doesn't need to be a way forward when the software 'just works' on every platform, I'm happily using it from my phone now.
https://github.com/ctoth/qlatt