As I expected, some folks reject the validity of these controlled tests, stating that different mics respond to different positioning in different ways, and their performance in their optimal position is the important issue. I can only suggest that they try some controlled testing in those different positions. After all, if the difference really exists, it should be apparent when levels and positions are matched, right?

And as I expected, some people pointed out that mics with different patterns and mics with very different transducer technology, like ribbon mics, sound different from the cardioid condensers I used. I absolutely agree.

Also as I expected, some people suggested that my use of a single source, the acoustic guitar, is preventing me from hearing the differences, which show themselves on cymbals and vocals. As Dirty Harry was wont to say, “A man’s gotta know his limitations.” I don’t record those sources so I don’t use them for testing .

One comment that has come up a couple of times is that the mics I chose were too similar, all large diaphragm mics from the low end of the spectrum. So I pulled out my highest priced mic, a Schoeps CMC64 small diaphragm condenser, and stuck it in the array.

Variety is the Spice of Mics?

For a little more variety, I replaced the Rode and AT mics with the Shure KSM141 and KSM44. So this set of clips includes two large diaphragm mics and two SD mics, and the price spread ranges from under $200 to over $1500.

I followed the same procedure I illustrated in yesterday’s post, aligning mic diaphragms by eye, playing a test tone through a small speaker about 1 foot from the mics, then adjusting gain on the M-Audio Profire 2626 to match their levels as well as possible in Reaper. Then I recorded the test tone followed by the acoustic guitar clip.

Mics and Clips

So here are the mics:

CAD M179
Schoeps CMC64
Shure KSM44
Shure KSM141

and here are the clips:

download 20090626-F.wav
download 20090626-G.wav
download 20090626-H.wav
download 20090626-I.wav

As before, I’ll post the key to the clip identities in a future blog entry. Or if you post your opinion and preference in a comment here or on one of the forums I visit I’ll email or PM the information to you.

Philosophical Musing

Please let me take a moment to wax philosophical here. No one has reported doing double blind ABX comparisons on the clips so far. Several posters have offered opinions on the sonic qualities of the clips, but no one has confirmed that they can actually hear a difference! This seems to be human nature, but I think that skipping the double blind check is missing a really valuable learning experience.

I (naively) expected that people would embrace an opportunity to do some controlled testing, using an easy but very effective comparison technique. I have found that careful ABX testing, using Foobar2000 and the ABX utility included in the program, has made me a better, more careful listener. I learned that the perceived differences between clips became much less when I no longer knew the source of the clip. As a result, I learned to search out subtle differences in tone color and texture. I also learned that I can’t hear any difference between systems that I have been told should show night and day differences.

I’d hoped that a number of other folks would try careful ABX testing of these clips with a statistically significant number of trials, so I could compare my results to theirs. At this point, I don’t know if my inability to hear the differences is normal or unusual.

The Question

If you visited here, listened to the clips, maybe even downloaded and installed Foobar2000, why did you not go the next step and run a test of 20 comparisons? Foobar2000 makes it easy to save your results and share them with the world, or not. Wouldn’t you like to have a personal evaluation of the difference between a high end preamp and a commodity unit, or between 44.1 and 192 sampling rates? Wouldn’t you like to contribute to the knowledge of the recording community?

After all, if the huge differences we read about in magazines and online are true, it will be easy to pick out the different samples, and we can get busy saving up for high end equipment. But if those differences are actually imaginary, driven by normal human traits like confirmation bias, we can save a bunch of money and time and trouble by ignoring gear lust and concentrating on mic placement and room acoustics.

So the question is, what do you have to lose by conducting a thorough series of ABX comparisons and reporting the results?

The Answers

Here are the sources for the clips from the last post:

090420Test1A.wav – Lynx & Hardy
090420Test1B.wav – Profire 2626

090420Test2A.wav – Profire 2626
090420Test2B.wav – Lynx & Hardy

090420Test3A.wav – Lynx 192
090420Test3B.wav – Profire 44.1

I hope these clips provided some useful information for you. They certainly helped me feel confident that I can make recordings with the Profire 2626 that will fully measure up to those made with the LynxTwo.

Since my interest in recording began only a few years ago, I’ve always had the internet as a resource for learning about the subject, I researched in every forum and magazine site I could find. And I now firmly believe that most of what I learned there was incorrect.

I should have been on my guard, because years ago in my pursuit of the playback side of audio I learned that uncontrolled listening tests are simply delusion at work, and that people routinely hear remarkable differences where there are none at all. But when I started trying to learn to record I was persuaded that different preamps and different a/d converters would make a night-and-day difference in my recordings. So I upgraded, then I upgraded again. When I started doing careful comparative listening, I realized that I wasn’t hearing these predicted major differences. In fact, I wasn’t hearing any difference at all.

Controlled Listening Tests

Since then I’ve tried to set up carefully controlled tests to compare gear. It’s not easy, at least for me. I seem to often miss some important detail in the setup, creating differences that shouldn’t be there. When I tried to compare three mic preamps I had the high pass filter (a low cut switch, in other words) active on one preamp. And when I tried to compare several field recorders, one recorder was set to record mp3s instead of waves, and once again the high pass filter was on. But I keep trying, and I’m getting a little better, I think.

ABX and foobar2000

ABX testing is a well established method for comparing two audio files (or other sources). A proper ABX test has only two items under test. The listener can take as long as they want, listen to either clip as many times as they want, go back and forth from the unknown X to the known A or B as often as they want. Then they state whether X is A or B. Not which they prefer, but simply which is which. Then the test is repeated for enough trials to achieve statistical validity.

ABX was originally hardware based, complicated, and expensive. But if we limit our testing to existing audio files we can do ABX testing in software. Various programs that implement ABX testing of digital audio files has been around for a number of years. The orginal PCABX.COM site has been allowed to lapse, but some of the introductory material is still available here.

I found a nifty program that makes the ABX process technically very easy. foobar2000 is a terrific freeware audio player that includes an ABX utility.

A New Audio Interface

I’ve been happy with my LynxTwo-C audio interface for a number of years. It has worked reliably, Lynx Studio has kept the drivers up to date and solid. But I’ve done a couple of sessions lately that could have used more inputs and more mic pres. The Lynx card offers some high powered expansion options, but I was also looking for a system that would integrate my monitor and headphone outs. I’ve been using a system that can only be called a kludge, although a successful one.

Meanwhile, the word on the M-Audio Profire 2626 has been good, I found a B-stock unit on Ebay and bought it. I began by installing the Profire on a nearby computer, leaving the Lynx card in my audio system. And with both systems installed, it was clearly time to try to do some carefully controlled listening tests.

Dynamic Mic, Two Preamps, Two A/Ds

Small variations in volume can apparently be recognized, but the listener hears a quality difference rather than a volume difference. Richard Clark has conducted hundreds of blind tests of amplifiers and says that he adjusts volume to .01db accuracy, although most people can’t detect differences of .1db.

In my first test of two preamps into two different converters, in an effort to create files of equal volume, I started each file with a test tone, generated from Adobe Audition and played through the LynxTwo output. I adjusted the John Hardy M-1 and the Profire input gain to create a signal at -18db, measured by eye on each system software mixer. Then I left that gain setting for the musical recording. I had planned to make the final precise adjustment to the gain in Adobe Audition, but to my surprise the software was precise only to .1db. So in spite of my efforts, the samples are at slightly different levels.

In this test I used a dynamic mic, an Electrovoice RE15 connected through a Coleman Audio LS3, basically just a y-connector, to the two preamps. I recorded my solo acoustic guitar about 2 feet (.6 meters) from the mic. This resulted in a very low signal and a tough test for the Profire preamp.

Here are a couple of clips that are easy to tell apart. I had hoped that the preamps on the M-Audio Profire 2626 would replace my faithful John Hardy M-1, but if you listen to the end of these clips you’ll hear a lot more noise in one clip – that’s the Profire.

download 090420Test1A.wav
download 090420Test1B.wav

But what if we trim off the end of the clip. Can you still tell the two recording chains apart?

download 090420Test2A.wav
download 090420Test2B.wav

Condenser Mic, One Preamp, Two A/Ds

I’ve seen quite a few debates about the audibility of a/d converters. Many people posting on the internet state as fact that prosumer level converters can’t compare with high end devices. And many prefer the sound of recordings made at high sample rates, insisting that they sound better even after conversion to the CD standard 44.1/16 format.

This time I used a Rode NT2a into the the John Hardy M-1, then the Coleman LS3 to split the signal to the line inputs of the LynxTwo and Profire 2626. Even with both units at nominal line level (+4dbu) there were small volume differences. Surprisingly, the unit with the longer cable run was louder. So once again we have slightly different volume levels that may make our ABX testing less valid.

download 090420Test3A.wav
download 090420Test3B.wav

How To ABX

Start by ownloading the clips above. Save them in a place you can find, like your music folder or your desktop. Maybe create a folder for this project.

Then you’ll need to download and install foobar2000. When you are prompted for the kind of install, choose Full.

When installing foobar2000, select the Full option

Start foobar2000 and open a pair of the test clips. The clips are named Test1A and Test1B, etc. Select both clips, right click, choose the Utils menu item, and there you’ll find ABX. Here’s a video that demonstrates the use of foobar2000 and its ABX comparator:


ABX Audio Testing with foobar2000 from Fran Guidry on Vimeo.

I hope some of you will download these samples and foobar2000 and conduct your own test. I’d be interested to hear the results of any ABX tests you conduct. Please contact me through the comments section with your results. I’ll post the keys to the samples in a future update. Let’s say, 2 weeks after this entry. (Mean, huh?)

I also hope you’ll make your own controlled comparisons and do your own ABX testing of preamps, converters, DAWs, cables, and other odds and ends of audio gear. Perhaps we can all learn something.