Audio over the Short-Time Fourrier-Transform

William Greenwood - Contact: greenwoodw50 [at] gmail [dot] com

Abstract

Audio over STFT uses a camera as well as a display to allow interaction with a sound. Sound is converted to a lossless spectrogram before being displayed to a screen. It can then be captured by the relevant capture device and converted back into sound. This process is very slightly lossy due to the compression of converting 16-bit integers into unsigned 8-bit integers.

Technical details

Audio samples are chunked together and a FFT is performed, this converts the samples into a complex array. The length of these vectors is the amplitude of the sound and is currently mapped to the value of the colour. The angle of the vector relative to the real axis is the phase of the sound and is usually discarded, this is mapped to the saturation of the colour.

The hue is currently unmapped but could potentially be used as error-correction data or a copy of either phase or amplitude.