I (naively) expected that people would embrace an opportunity to do some controlled testing, using an easy but very effective comparison technique. I have found that careful ABX testing, using Foobar2000 and the ABX utility included in the program, has made me a better, more careful listener. I learned that the perceived differences between clips became much less when I no longer knew the source of the clip. As a result, I learned to search out subtle differences in tone color and texture. I also learned that I can’t hear any difference between systems that I have been told should show night and day differences.

I’d hoped that a number of other folks would try careful ABX testing of these clips with a statistically significant number of trials, so I could compare my results to theirs. At this point, I don’t know if my inability to hear the differences is normal or unusual.

The Question

If you visited here, listened to the clips, maybe even downloaded and installed Foobar2000, why did you not go the next step and run a test of 20 comparisons? Foobar2000 makes it easy to save your results and share them with the world, or not. Wouldn’t you like to have a personal evaluation of the difference between a high end preamp and a commodity unit, or between 44.1 and 192 sampling rates? Wouldn’t you like to contribute to the knowledge of the recording community?

After all, if the huge differences we read about in magazines and online are true, it will be easy to pick out the different samples, and we can get busy saving up for high end equipment. But if those differences are actually imaginary, driven by normal human traits like confirmation bias, we can save a bunch of money and time and trouble by ignoring gear lust and concentrating on mic placement and room acoustics.

So the question is, what do you have to lose by conducting a thorough series of ABX comparisons and reporting the results?

The Answers

Here are the sources for the clips from the last post:

090420Test1A.wav – Lynx & Hardy
090420Test1B.wav – Profire 2626

090420Test2A.wav – Profire 2626
090420Test2B.wav – Lynx & Hardy

090420Test3A.wav – Lynx 192
090420Test3B.wav – Profire 44.1

I hope these clips provided some useful information for you. They certainly helped me feel confident that I can make recordings with the Profire 2626 that will fully measure up to those made with the LynxTwo.

Since my interest in recording began only a few years ago, I’ve always had the internet as a resource for learning about the subject, I researched in every forum and magazine site I could find. And I now firmly believe that most of what I learned there was incorrect.

I should have been on my guard, because years ago in my pursuit of the playback side of audio I learned that uncontrolled listening tests are simply delusion at work, and that people routinely hear remarkable differences where there are none at all. But when I started trying to learn to record I was persuaded that different preamps and different a/d converters would make a night-and-day difference in my recordings. So I upgraded, then I upgraded again. When I started doing careful comparative listening, I realized that I wasn’t hearing these predicted major differences. In fact, I wasn’t hearing any difference at all.

Controlled Listening Tests

Since then I’ve tried to set up carefully controlled tests to compare gear. It’s not easy, at least for me. I seem to often miss some important detail in the setup, creating differences that shouldn’t be there. When I tried to compare three mic preamps I had the high pass filter (a low cut switch, in other words) active on one preamp. And when I tried to compare several field recorders, one recorder was set to record mp3s instead of waves, and once again the high pass filter was on. But I keep trying, and I’m getting a little better, I think.

ABX and foobar2000

ABX testing is a well established method for comparing two audio files (or other sources). A proper ABX test has only two items under test. The listener can take as long as they want, listen to either clip as many times as they want, go back and forth from the unknown X to the known A or B as often as they want. Then they state whether X is A or B. Not which they prefer, but simply which is which. Then the test is repeated for enough trials to achieve statistical validity.

ABX was originally hardware based, complicated, and expensive. But if we limit our testing to existing audio files we can do ABX testing in software. Various programs that implement ABX testing of digital audio files has been around for a number of years. The orginal PCABX.COM site has been allowed to lapse, but some of the introductory material is still available here.

I found a nifty program that makes the ABX process technically very easy. foobar2000 is a terrific freeware audio player that includes an ABX utility.

A New Audio Interface

I’ve been happy with my LynxTwo-C audio interface for a number of years. It has worked reliably, Lynx Studio has kept the drivers up to date and solid. But I’ve done a couple of sessions lately that could have used more inputs and more mic pres. The Lynx card offers some high powered expansion options, but I was also looking for a system that would integrate my monitor and headphone outs. I’ve been using a system that can only be called a kludge, although a successful one.

Meanwhile, the word on the M-Audio Profire 2626 has been good, I found a B-stock unit on Ebay and bought it. I began by installing the Profire on a nearby computer, leaving the Lynx card in my audio system. And with both systems installed, it was clearly time to try to do some carefully controlled listening tests.

Dynamic Mic, Two Preamps, Two A/Ds

Small variations in volume can apparently be recognized, but the listener hears a quality difference rather than a volume difference. Richard Clark has conducted hundreds of blind tests of amplifiers and says that he adjusts volume to .01db accuracy, although most people can’t detect differences of .1db.

In my first test of two preamps into two different converters, in an effort to create files of equal volume, I started each file with a test tone, generated from Adobe Audition and played through the LynxTwo output. I adjusted the John Hardy M-1 and the Profire input gain to create a signal at -18db, measured by eye on each system software mixer. Then I left that gain setting for the musical recording. I had planned to make the final precise adjustment to the gain in Adobe Audition, but to my surprise the software was precise only to .1db. So in spite of my efforts, the samples are at slightly different levels.

In this test I used a dynamic mic, an Electrovoice RE15 connected through a Coleman Audio LS3, basically just a y-connector, to the two preamps. I recorded my solo acoustic guitar about 2 feet (.6 meters) from the mic. This resulted in a very low signal and a tough test for the Profire preamp.

Here are a couple of clips that are easy to tell apart. I had hoped that the preamps on the M-Audio Profire 2626 would replace my faithful John Hardy M-1, but if you listen to the end of these clips you’ll hear a lot more noise in one clip – that’s the Profire.

download 090420Test1A.wav
download 090420Test1B.wav

But what if we trim off the end of the clip. Can you still tell the two recording chains apart?

download 090420Test2A.wav
download 090420Test2B.wav

Condenser Mic, One Preamp, Two A/Ds

I’ve seen quite a few debates about the audibility of a/d converters. Many people posting on the internet state as fact that prosumer level converters can’t compare with high end devices. And many prefer the sound of recordings made at high sample rates, insisting that they sound better even after conversion to the CD standard 44.1/16 format.

This time I used a Rode NT2a into the the John Hardy M-1, then the Coleman LS3 to split the signal to the line inputs of the LynxTwo and Profire 2626. Even with both units at nominal line level (+4dbu) there were small volume differences. Surprisingly, the unit with the longer cable run was louder. So once again we have slightly different volume levels that may make our ABX testing less valid.

download 090420Test3A.wav
download 090420Test3B.wav

How To ABX

Start by ownloading the clips above. Save them in a place you can find, like your music folder or your desktop. Maybe create a folder for this project.

Then you’ll need to download and install foobar2000. When you are prompted for the kind of install, choose Full.

When installing foobar2000, select the Full option

Start foobar2000 and open a pair of the test clips. The clips are named Test1A and Test1B, etc. Select both clips, right click, choose the Utils menu item, and there you’ll find ABX. Here’s a video that demonstrates the use of foobar2000 and its ABX comparator:


ABX Audio Testing with foobar2000 from Fran Guidry on Vimeo.

I hope some of you will download these samples and foobar2000 and conduct your own test. I’d be interested to hear the results of any ABX tests you conduct. Please contact me through the comments section with your results. I’ll post the keys to the samples in a future update. Let’s say, 2 weeks after this entry. (Mean, huh?)

I also hope you’ll make your own controlled comparisons and do your own ABX testing of preamps, converters, DAWs, cables, and other odds and ends of audio gear. Perhaps we can all learn something.

Even though it’s not strictly necessary, it’s interesting to use some kind of acoustic measuring tool to evaluate the room and gauge the results of treatment. I recently learned of a free program that works with a PC or Mac and their audio systems to measure and display room response. It’s called Room EQ Wizard and it’s available for download at the Home Theater Shack. The Shack is a discussion forum focusing on home theater (well, duh), but Room EQ Wizard, or REW as they call it, is just as useful for recording spaces as it is for home theater.

The home theater users seem to have systems that are a bit more complex than my simple recording rig. I don’t have a subwoofer and accompanying crossover, nor do I have an equalizer in my playback chain. So my connections were simpler than those illustrated in the REW help pages.

I bought one extra piece of equipment for this project, a Sound Pressure Level (SPL) meter. These are available from Radio Shack and other sources. The Radio Shack unit I bought is accurate enough to measure a baseline volume setting for all my readings. It’s possible to use the SPL meter as the measurement mic, but it is not accurate above about 3Khz, fine for tuning subwoofers but not much help for a full range monitor system. The Behringer ECM8000 is widely recommended for an inexpensive measurement mic, but I have a couple of DPA 4061 miniature omni mics in my collection, so I chose that for my measurement mic.

The first step in using REW is a soundcard calibration procedure. This procedure compensates for nonlinearities in the soundcard and also ensures that the basic system is working, with REW finding the soundcard inputs and outputs as needed.

I use a LynxTwo-C soundcard and REW found it easily. Here’s the Settings page where the soundcard is configured and measured.

The soundcard calibration starts by creating a loopback connection, that is, the output of the audio interface is connected to the input. This is easy with a recording oriented system, where balanced line in and line out connectors should be available. On the Lynx breakout cable the XLRs just clicked into place. On other systems a male TRS to male TRS may be needed. Don’t forget to turn off your power amp or mute your speakers – now how would I know to remind you about that??

I’ve created a YouTube account for videos related to the blog. You might stop by http://www.youtube.com/user/homebrewedmusic if you’re in the neighborhood. This video goes through the steps to run the calibration measurement.

After saving the soundcard calibration file we need to adjust the levels so we’re getting a good signal to noise ratio but avoiding clipping. First we restore the loopback connection to our normal hookup and turn on our power amp or unmute our speakers.

This video demonstrates the steps:

Since we’re moving these input and output volumes all the time we’ll probably need to run this level adjustment routine before taking measurements.

With the levels set we’re ready to measure our room/speaker response. Well, actually we’re also including the mic preamp and power amp, but those are probably quite linear, especially compared to our room and speakers.

Here’s a screen video of the measurement process.

And here’s the resulting graph. Looks pretty ragged to me, with huge swings between 40 hz and 200 hz, and lots of comb filtering in the higher frequencies. I suspect that this is the normal response of a medium small room. In our next entry we’ll see what we can accomplish with our current batch of broadband absorbers.