Automating audio testing in consumer devices
20 April 2015
With so much audio being consumed through streaming and downloads, customers expect perfect sound from their laptops, tablets and phones. Yet digital audio subsystems have their own unique challenges for the designer and system integrator - not least validation.
Audio validation presents special problems in production engineering. Automation is key to throughput, but any analogue system is hard to analyse swiftly and precisely in bulk. Audio is particularly tricky, as listening tests are time-consuming, fatiguing and too noisy to conduct in high density lab environments. Automated testing risks missing audible artifacts and frequently needs modification to the devices under test, or bypasses some part of the end-to-end reproduction or capture chains.
Wipro faced all these problems recently when building a SoC-based consumer tablet system for an OEM, and evolved a solution which, while not perfect, proved extremely useful in increasing the quantity and quality of audio testing during production.
Before they adopted the new process, they were manually running some 4000-plus test cases in an acoustic room as part of the audio driver validation process, which when system set-up and test vector selection were taken into account needed 24 working hours and 15 engineers to complete. As the process needed intensive listening and manual manipulation of large spreadsheets, the testers quickly got tired and frustrated so quality suffered and errors increased. It was by far the most complex and least satisfying part of product verification.
The platform they were designing for had a hardware certification kit which was based on a client-server model. For most of the non-audio systems, the devices under test were loaded with the client and connected to the network. The server detected the client, connected over the network, loaded on a set of tests, ran them and gathered the results automatically. The process was well understood, reliable, and swift and already in place: how could it be adapted to replace the listening tests?
Wipro designed three components: The first was the audio client which ran the sound system through the test cases automatically, setting a range of modes, audio routing, switching, and volumes. The second was an analogue loop-back connector, which used the audio system’s own audio input to sample its output. That digital output stream was then sent in real time to the third component, a Fast Fourier Transform (FFT) software module running on the client, which produced numerical measurement of frequencies and levels of audio energy detected by the system.
Although the tests only used pure sine source files - generated mathematically using the open-source audio manipulation tool Audacity - any delay or glitch in audio processing in input or output produced characteristic FFT results that could be automatically tested for. In general, nonlinearity in time or frequency domain has a distinct fingerprint that reliably colours single-frequency sines. The FFT calculations are not complicated - Wipro used the open source tool Kiss FFT - and the test results, compiled into XML format files, can be used exactly as any other produced by the hardware test framework. The result is what was expected, audio testing included as a standard part of the automated process.
Wipro found this worked really well, detecting glitches that were inaudible through manual testing, Other major advantages are that all the tools required are license-free, the process neither generates nor is sensitive to audible noise, it avoids any need to compare results with ‘golden reference’ test files - a process needing large amounts of memory and disk space, and it is highly parallelisable. Tens or more devices can be simultaneously tested on one bench, provided only that they’re networked to the test system server. There’s no need to alter the devices being tested, and the entire audio chain is part of the process - unlike, say when software loopbacks are used.
It’s also very easy to operate, no harder than any other system test, so has the potential to be made part of larger production testing by less-skilled workers
The limitations are that it does only work with single frequency sine test vectors, although these can be presented in any format the tested platform supports, and that some actions still require manual input. These include actions like registering and pairing Bluetooth headsets and more complex audio systems with multiple outputs would also require specialist attention.
However, the approach is very flexible and could be used for a wide variety of consumer devices that include digital audio. Technically, testing across a large range of discrete sine frequencies and levels exercise the audio chain well; there are few if any deficiencies in consumer digital audio that do not reveal themselves to this class of testing, even if more complex test material has traditionally been used for human-based analysis. Digital FFT is sensitive to both hardware and software issues.
Wipro’s experiences were satisfactory across the board. The most immediate impact is on time for testing - from the 24 hours total of the old method, they could run a system through its paces in three to four hours, and with far greater reliability and reproducibility. They could also run as many systems through the tests at the same time as they wanted, within reason: if they needed to check two or three possible solutions to a problem, it took no more time than testing just the one.
The results are better than human-mediated testing, the approach is very economical to implement, and the concept can be adapted to many different sorts of audio-enabled devices. As the tools used are open source and the loop-back cable simple, cheap and flexible, the procedure should be easy to integrate into any automated test framework that’s designed to work with general purpose hardware, and very feasibly adaptable for more specialist uses.
The customer is happy because design changes are quicker to validate and the quality of the end product is higher. The engineers are happier because there are far fewer frustrating, fatiguing tests that delay progress and make poor use of their skills, and company managers are happier because Wipro have moved another step closer to the ideals of continuous design and delivery, using automation to industrialise the process. Lastly, the most important stakeholder - the end consumer - is happier, because they get better, cheaper, timelier products.
Contact Details and Archive...