0
Spectral processing is a technology we use and tout in our flagship plugins, but how does it work? And what are the benefits of this audio technology?
Note: We replaced smart:EQ 3 with smart:EQ 4! The intelligent equalizer now spectrally balances both single tracks and entire mixes and sports auto gain, dynamic standard filters,… and much more.
When digital technology got fast enough to process audio easily, the cool new thing was multiband processing. Instead of having a tool running a set of processes on your whole signal – a compressor, for example – we were able to split the signal into bands, usually four, and define different settings for each area of the frequency spectrum. We got more control and better results.
Today’s processors can handle more than just a multiband setup, though. Theoretically, we’re able to process hundreds, thousands of bands at once. But with this scaling comes a problem: setting it up. It might take longer to program a multiband compressor than a single-band one, but it still doesn’t take forever. Programming two thousand bands, though? We’re going to need a new system.
Fortunately, in making spectral processing systems, defining how to manage and control parameters for many bands is part of the job – and that job is usually down to the developer rather than the end user.
There’s a way to have your cake and eat it too: to make more bands easier to program than a four-band multiband setup. If a spectral processor’s many bands are governed by a particular rule, then any given band should have the right parameters to behave appropriately. The drawback is that it requires more effort on the part of the developer.
At sonible, our spectral processing effects have been tuned and refined to give a different way of working in each application. Each effect – whether it’s smart:comp 2, smart:limit or others – plays host to a finely customized setup of bands under the hood to get its processing done, and to increase the sound quality over effects of the past. They all do their thing differently, but the principles are the same.
So how does spectral processing work behind the scenes?
The human hearing system is at the heart of how we listen to music, so shouldn’t it be at the heart of the tools we use to make music? Spectral processing has exactly this philosophy behind it.
Audio is usually computed in small chunks of time, but our hearing system hears in the frequency domain. Spectral processing matches our ears by first splitting the input signal into many different ‘bins’ using a Fourier transform.
These bins are spaced in a linear way – for example, with a spacing of 10Hz, you’d get one bin covering the frequencies between 30Hz and 40Hz, but also a single bin covering the frequencies between 1,010Hz and 1,020Hz. The next thing we do is to ‘group’ multiple bins into bands, which is what will actually be detected, processed, computed, analyzed or whatever is required by the plugin in question.
So our bands are derived from our bins, but how do we choose which bins go into making which bands, and where?
Specifically, exactly how the bands are laid out across the frequency spectrum depends on the application, but there are some general rules of thumb: higher frequencies are less easy for our ears to resolve usually requiring fewer bands to get the results we desire; and with low frequencies it’s usually the opposite. Also, our hearing system is almost always particularly sensitive to frequencies between about 100Hz and 500Hz – the range of the fundamental frequency of the human voice and many instruments. You’d expect to find a higher resolution of bands in this region in most sonible spectral processing.
Even the input signal you’re using might cause the bands in a sonible spectral processing plugin to change. Depending on the profile you’ve selected or the audio that’s been analyzed by the plugin, it may become primed to listen out for different frequency areas with more or less sensitivity.
And then there’s the choice of how to map the bands. How should each band’s signal be weighted for its amplitude at different loudnesses? Should bands be separated by a psychoacoustic scale such as the Mel scale or Bark scale? Once again, it depends on the application – a compressor and a reverb may require very different arrangements.
As we’ve been mentioning from the start, a fundamental advantage of spectral processing technology is that it works in the same way as the human hearing system, breaking sound up for processing into different frequency ranges, rather than applying calculations directly to a sound’s waveform in the time domain.
Since all its audio manipulation happens within the frequency domain, spectral processing is a great analog for how we actually interpret and hear sound. We can get higher-resolution at critical points in the frequency range, and lower-resolution where it matters less due to either our hearing system or the processing we’re hoping to perform.
Spectral processing also means that we can ‘reach into’ a piece of audio, removing or changing things that are happening at particular frequencies without affecting everything else.
But we have to be careful when using the power of spectral processing. When ‘reaching into’ a sound in this way, it can be possible to remove something you didn’t intend to, and so spectral processing systems need to be built very carefully.
To illustrate how spectral processing all comes together to create a plugin, here are some real-life examples of the technology within the sonible range.
We’ll start with smart:EQ 3, which can use more than 100 bands in order to analyze instances between tracks and make decisions about how each one’s EQ should be behaving.
When it comes to smart:reverb, a field of approximately 30 bands was enough to get great results out of generating a reverb profile that doesn’t clash with the input signal. In each plugin, we find the optimum balance between number of bands and processor load, while making sure the quality of results doesn’t suffer at all.
Moving onto smart:comp 2, if you could lift its lid, you’d find over 2000 bands at work in its spectral processing. The number ensures that any transient spikes across the frequency spectrum can be detected, and also keeps bass response fully monitored and completely under control. smart:comp 2’s detection processing actually requires even finer band resolution than the gain reduction circuit that applies the results of that analysis.
Spectral processing has emerged as a viable, effective and creative solution in music production over the last few years. At sonible, we’re glad to have pioneered it as a forward-thinking technology, supported by our AI for many use cases.
As other developers realise the power of spectral processing and promote new uses of and ideas for the technology, we’re looking forward to what comes next and will be working on new directions of our own.