HN Explorer

Ask HN: Where are the good resources for learning audio processing?

164 points by SamCoding ↗ HN
I'm trying to program a harmonizer, like Jacob Collier's one built by MIT's Ben Bloomberg. I am looking for good, accessible resources on pitch shifting (whilst still sounding natural) and other terms I've heard like format shifting.

Where are some good resources for this for somebody with extensive programming experience but no experience in audio processing?

50 comments

[ 2.6 ms ] story [ 80.4 ms ] thread
i_am_proteus
https://ccrma.stanford.edu/~jos/

Not much to say that Julius doesn't... open course materials for (almost) everything you might need in audio processing.

planewave
Second anything from CCRMA, the inventors of FM synthesis and still the one top programs in the country/world.
worstspotgain
Thirded! CCRMA and its people are awesome (way more so than the rest of Stanford.)
anymouse123456
This comment helped get me over the barrier to take a closer look...
geokon
I've used it as a reference or to gain a different perspective on something I'm familiar with - but it's generally way too terse to learn from

Ex: https://ccrma.stanford.edu/~jos/mdft/Bessel_Functions.html

worstspotgain
That doesn't look terse to me, though it does require familiarity with the subject.

"The last expression can be interpreted as the Fourier superposition of the sinusoidal harmonics of [expression], i.e., an inverse Fourier series sum. In other words, [expression] is the amplitude of the k-th harmonic in the Fourier-series expansion of the periodic signal x_m(t)."

Many of the concepts are hyperlinked for reference. With the required familiarity, I would much rather read this than something that took seven pages to get to the point - say by assuming that the reader is unfamiliar with a premise out of an abundance of caution.

cozzyd
Julius is a national treasure. I learned immensely from his class and textbooks...
Chrupiter
When I wanted to make a python application to separate a song into the source instruments I used this: https://www.coursera.org/learn/audio-signal-processing. I studied signal processing as a Computer Engineer student but I didn't really get it at the time, with that course I understood what I could do in practice.
aj7
Use LabView as a calculation engine to do experiments. The advantage is you get system-like diagrams.
bwestergard
You should learn Supercollider, starting with Eli Fieldsteel's tutorials.
an_aparallel
Hey - one of the industry standard time stretching library is "elastique" by Zynaptiq (licensed, not open source). Used by Ableton, FL Studio etc.

If you want to peak into some source code - you can look into Rubberband library:

https://breakfastquay.com/rubberband/

Rubberband is one of the time stretching/pitch shifting algorithms used in Reaper. You can download reaper trial and listen to the results with different parameters to see how you can tweak the code and if that gets any results you're happy with:

https://www.reaper.fm/

IndySun
>Hey - one of the industry standard time stretching library is "elastique" by Zynaptiq.

The company is Zplane, not Zynaptiq. Easy mistake, there is a little overlap.

https://licensing.zplane.de/

planewave
Professor Puckett, inventor of Max and PureData (the two top visual programming languages for DSP) has a book, The Theory and Practice of Electronic Music, with interactive examples written in PD, this one probably has an example exercise for a pitch shifter [0]

I often recommend also Music and Computers originally out of Columbia. [1]

[0]http://msp.ucsd.edu/techniques.htm

[1]https://musicandcomputersbook.com/

bwanab
Very cool. I've used PD a lot in the past, but I didn't know about his book!
a-dub
he gives awesome talks as well!
fxtentacle
Audio is half art, half science. That's why I'd try to find someone with experience.

Back in university, I heard lectures on FFT and its applications to audio signal processing. So open access university courses would be the second place I'd look. The approach I always try first is to ask people I know if they can recommend a conference/meetup. For example, the annual JUCE events appear to be chock full with VST plugin developers. There's also private schools like SAE where you (or your employer) can pay for you to have an hour with one of their lecturers to ask questions.

the__alchemist
I would pick up a microcontroller dev board that has a mic built in (Eg one of the STM32 discoveries). Also get a "codec" dev board. (Or alternatively, use the MCU's onboarod DAC). Get it to receive audio, process it using DSP, then output it, and/or save to memory. This will really force you to understand it.
p1esk
Why not just use a regular laptop for this? There’s a ton of low level sound processing libraries for every OS.
spacechild1
Bad advice. I have no idea how using a microcontroller would help someone understand pitchshifting algorithms.
dsp_person
I find [1] a good reference. A con is the examples are in matlab, but it's clear enough between the text and matlab code to write your own implementation.

Also [2] is a decent book for overall dsp concepts.

[1] DAFX - Digital Audio Effects (Second Edition) Edited by Udo Zölzer https://dafx.de/DAFX_Book_Page_2nd_edition/index.html

[2] Understanding Digital Signal Processing, Richard Lyons

cushychicken
Zolzer’s book is the best out there on the topic that I know of.
a-dub
lyons is a good intro but maybe a bit handwavey at times (although my copy is an edition from the 90s).

consider maybe backing it up with one of the textbooks like oppenheim (the classic) or manolakis (one that i think i remember liking).

cageface
The Will Pirkle books have a lot of good info and code to get you started:

https://www.willpirkle.com

Audio programming is a lot of fun but it's the most challenging domain I've ever worked in. You have to be very careful with what you do on the audio thread. No locks, no memory allocation etc. Messing this up can result in some really ugly audio artifacts.

mikewarot
Gnu Radio can easily handle audio I/O as well as it does IQ signals from SDR front ends. It's cross platform and you just build flow graphs, which then can be executed.
Ylpertnodi
www.airwindows.com may help.
squeaky-clean
Chris @ airwindows is super nice to release all his plugins and source code for free. But the code quality is really bad. That doesn't matter if you're using a plugin in the production of a song and it works well. But for learning dsp, it's a bad resource.
mottosso
I was curious what you meant and went to have a look. At first all seemed well, until I got to the actual audio processing part. :)

https://github.com/airwindows/airwindows/blob/master/plugins...

Then again, maybe this is the norm for audio engineers? Not my field.

al2o3cr
The documentation for that specific module even calls it out as "painfully hard-coded biquad filter code", YMMV.

I'd guess those files aren't what the author actually edits - there are commits that suggest that they are _generated_ from "boilerplate", and even a few files that seem to have failed interpolations ("__MyCompanyName__" in some copyright lines, for instance)

A lot of files also seem to have duplicated code, down to individual comments. For instance, the comment on line 24 reoccurs on line 344 of this effect:

https://github.com/airwindows/airwindows/blob/master/plugins...

and in the Mac AU version on line 267:

https://github.com/airwindows/airwindows/blob/master/plugins...

and in the Linux VST version on lines 24 and 344:

https://github.com/airwindows/airwindows/blob/master/plugins...

djaychela
Not looking for an argument, but can you give some pointers as to what is bad about the code?
squeaky-clean
Someone already shared an example of how difficult the dsp code is to read. There's tons of magic numbers and short variable names with no description of why the code does what. Again, that's cool if you want to use the compiled version in a song, but it's not friendly to learn from. But I don't think the goal of this project is to be educational, it's just a little bonus that the code is open source.

There's also just things like uneven indentation, lots of things to reduce linecount at the expense of readability like single line if statements. Old optimization tricks that aren't necessary with modern compiler like `while (--sampleFrames >= 0)`.

Here's another example file

https://github.com/airwindows/airwindows/blob/master/plugins...

And the project structure is really weird. Normally you would use a framework like JUCE or iPlug, or write your own, such that your dsp code is written once and your multiplatform code is separate. Instead every platform (Mac, Windows, Linux, VST, AU) are a separate codebase with all the dsp code duplicated.

He's definitely using some sort of templating system to do this, maybe even using some tool that lets him write the dsp in Python or Matlab and converts it to C++. Basically these files in the github are not his true "source", this is generated from his real source. For that reason, commits are often massive and unhelpful in tracing changes to any individual effect.

For example, a recent commit where he added the WolfBot effect, a guitar amp simulator, has 427 changed files and 150,284 added lines. Also even though the commit is titled WolfBot, this commit includes the addition of other fx, CreamCoat, DeRez3, kCathedral3, kGuitarHall, kPlate140, kPlate240.

https://github.com/airwindows/airwindows/commit/7623a1c14b01...

djaychela
Thanks, much appreciated that you took the time to point those things out.
MaysonL
Check out officehours.global – a lot of audio people hang out there.
bbloomberg
Hello :) all of the resources mentioned here are great! One step I’d add to the learning part (and it’s what we did when building Jacob’s) is to spend a lot of time trying out existing implementations to determine what you like and don’t like.

For example, many of them don’t have great low end. Some are “sluggish” and need external enveloping. Getting a sense for what’s out there can help to provide a North Star when you write your own. Some classics are the Eventide H3000, IZotope Vocal Synth, TC Voice Live, Antares Harmony Engine, and Soundtoys Little Alterboy.