A New ABX Tool

Wednesday, July 30th, 2014

At first glance evaluating audio gear seems like it should be perfectly simple, all we need to do is trust our ears. But it turns out that we don’t hear with our ears, we hear with our brain and with the nervous system circuitry between our ears and brain, all in a wildly complex network of sensing, filtering, focusing, and feeding back. And this network is both amazingly sophisticated and hilariously inept at the same time, able to discriminate between small differences but also certain to occasionally make up the answers.

The Brain Is the Problem

Here’s a YouTube video featuring James D. Johnston discussing issues related to human hearing and testing. The whole video is interesting, but this link starts at the point where JJ is explaining our current understanding of human perception. He covers many points that lead directly to the need for double blind testing for audio comparisons.

If we want real answers we need to work around the known issues with human perception, issues like confirmation bias, sense memory, attention. We need to hide the labels, switch between the test item and the comparison items easily and quickly, select a particular moment for focus. And we need to do enough trials that we have some statistical confidence in the result. Meeting these requirements can be quite a challenge when comparing hardware, but when we can resolve our questions with digital recordings we can do some serious testing with a free ABX software tool.

ABX Takes Tools

ABX testing is a simple concept but a powerful one. It’s not a tool for testing preference, its only purpose is to tell if the subject can hear a difference between two samples. The subject is presented with two known samples, A and B, along with one unknown, X. The unknown X is either A or B but is unlabeled. The subject controls the switching for A to B to X, listens for as long as they wish to each sample, then identifies X as either A or B. Then a new randomly selected X is presented and the test is repeated. With digital audio files a software ABX tool is just the ticket, and it’s straightforward for the program to maintain a record of selections and compute the accuracy and confidence level for the session.

This spring REAPER forumite clepsydrae (Casey Connor) started a thread offering a free Java based ABX tool. He has enhanced the program several times and the current version is very functional for my purposes. In fact this ABX tool adds an interesting twist, a Shootout function that presents multiple source samples for comparison.

ABXer in Action

When we start the Lacinato ABX program we open on this screen:

Opening Screen

The text gets us started with the basics of ABX. We can use a file open dialog by clicking on the A or B buttons or drag and drop our samples into the slots.

03--no-hold

Once the files are loaded we can play them by pressing the play button for that file. There are adjustments for gain(dBFS) and time offset(s) but my usual procedure would be to carefully match levels and start positions before bringing my sample files into the program.

Notice that I’ve turned off the Hold Pos. button. I nearly always want to start each comparison at the same point, while selecting this option continues to play through the sample as we switch between candidates.

I find it very useful to identify a portion of the samples where I think I hear the most difference. The play start position slider lets me select where the sample will start playing.

04--slider

This selection affects all the samples, so we can compare on a phrase or even one note easily.

Once we’ve identified an area where we hear a difference and set the slider to start there, we’re ready to begin our ABX process. When we press the Start AB*X button it changes.

005-start-abx

Now the button offers to Play X Again and this becomes our trigger to hear sample X. I often bounce around between A, B, and X to solidify the difference in my mind.

When I have a strong feeling that I know the identity of X I press the choose button to register my choice, and that brings the Play Next X button back.

006--next-x

After a reasonable number of trials, say 10 to 15, I’m ready to look at my results. In this example I just made wild guesses so my results are much worse than if I’d simply flipped a coin.

007-results

The Confidence value is the key measure and should be over 95% for a solid indication that there exists a difference that is audible to the subject. Even that level of confidence is not certainty, so the more trials and more subjects the better for critical testing.

Here’s a video demonstrating a session with the Lacinato ABXer:

Here are the same source level matched files I used in this demo:

Rode NT2a
Schoeps CMC64

More Than ABX

This program is called the ABX/Shootout-er because it goes beyond the two factor ABX comparator. The author offers it as a way to test your preference objectively. He has this to say in describing the feature:

Yes, and you can also use it to do shootouts: “i think i like this last mix i did better than the others, but maybe that’s just cause it’s the last one i did…” Load up the mixes into the program, start a shootout, pick your favorite, repeat a bunch of times, and it’ll tell you which you actually like the best. (Or the least, depending on the question you want to answer.)

I used the Shootout-er function to evaluate four mics I recorded for a recent blog post.

Shootout file screen

Here are all four files loaded. I selected them all in a Windows Explorer window and dragged and dropped them on the first line of the file list. The program moved one file to each slot automatically.

Then it’s simply a matter of pressing the Start Shootout button.

010--shootout-1

Now we see the four files shuffled for our evaluation. Listen to all four then choose a preference. In some cases this might be an ideal time to use the Hold pos. or Loop features.

011--shootout-2

Each choice is registered by the program and another shuffle is executed. When a reasonable number of tests has been made or the tester burns out from the effort, hit the Show Results button to see if the preference was real or imaginary, and which sample gets the nod.

012--shootout-3

Here’s a video demo of the Lacinato Shootouter in action:

Here are the four level matched same source clips I used in the demo:

r.wav
s.wav
t.wav
u.wav

Get Real (Information)

The reason I return over and over to these blinded comparisons is simple, bad comparisons are worse than no comparison at all, they give us bad information which our brains then lock on as solid gold truth. So we actually know less after testing rather than more. In order to really hear differences in a comparison we need to get rid of the influences that can so easily override the real sound, the real information. Thanks to Casey Connor at Lacinato.com we have a terrific new tool to help us get the real information.



This entry was posted on Wednesday, July 30th, 2014 at 12:28 pm and is filed under Audio, Tutorials. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

  • LAMPG Video in HitFilm Express 2017
  • Can Shotcut Cut a LAMPG Video?
  • -->

    4 Responses to ' A New ABX Tool '

    Subscribe to comments with RSS or TrackBack to ' A New ABX Tool '.

    1. MerseyMale said in post # 1,

      on April 13th, 2015 at 1:45 pm

      Yep! It’s my brain that’s the problem…
      Can I have a new one, please? Preferably with talent, this time?
      😉

    2. Fran Guidry said in post # 2,

      on April 14th, 2015 at 12:55 pm

      As far as human perception, all of us face the same limits. The key trick is to recognize those limits rather than ignoring them, and using tools like ABX to work around them.

      On the subject of talent, I think sometimes talent is overrated. My wife has a remarkable sense of rhythm, she learned to play a basic rock’n’roll backbeat in 15 minutes. But she never practiced so after 15 years she hadn’t progressed at all.

      I’m not very musically talented according to everyone I grew up with, including all my teachers and my Mom, but once I developed a passion for slack key guitar I busted my butt to learn how to make that sweet sound, and now people tell me they love my music.

      So instead of wishing for a brain with talent, wish for passion and determination, it will get you further I think. Of course a combination of all three is ideal, but let’s not get greedy (grin).

      Fran

    3. Eddy Bugnut said in post # 3,

      on June 5th, 2015 at 12:52 am

      I would love this tool but it keeps on crashing my system which is a pretty high end system. Can’t seem to figure out a workaround.

    4. Fran Guidry said in post # 4,

      on June 5th, 2015 at 9:36 pm

      Sorry, I don’t have any clue on how to improve the stability of the program. Did you contact the author? How are you executing the program?

      Fran

    Leave a reply






    About the Blog

      Howdy, my name is Fran Guidry and this is my Homebrewed Music blog.

      I play Hawaiian slack key guitar and recorded my solo acoustic CD at home. Most of the recording information I find on the internet seems focused on bands, drums, multitracking, and so on but my main focus is recording solo acoustic guitar. Lately I’ve been enjoying video recording along with audio, so that shows up in the blog as well.

      I’m also a guitar nut. I love big ones and little ones, handmades and factory guitars, cheap ones and expensive ones. So I’ll be sharing the fun of exploring guitars as well, along with the challenges of amplifying acoustic guitars for live performance.

      Welcome!

    Philosophy

      My recording philosophy is pragmatic, skeptical, not super critical. After all, the performance is by far the most important component of a track, and every aspect of any recording is a matter of taste.

      But I do like to know “about stuff.” Back in hifi days I learned about double blind testing. I learned that we humans can easily hear differences that don’t really exist. The more I’ve learned about our human auditory system, the more I’m skeptical of what people say they hear, especially if they claim that a particular microphone or preamp or cable has some magical property.

      I’ve only been recording since 2001, and when I started I found the usual places on the internet. I sought advice and accepted it, thought I would improve my recordings by using more expensive equipment. It didn’t work.

      Two things that did seem to lead to better recordings were experience and room treatment. Getting an appealing sound is the combination of many small details, and learning those details only comes from experience. Amd the sound of the recording space is obviously a big factor.

      I’ve only recorded seriously using digital technology, but I remember trying to record rehearsals and gigs back in analog days. I don’t have any nostalgia for analog recording and playback systems at all. I think even low end digital systems can capture marvelous recordings. So when I look at gear, I look for good specs: low noise, broad flat frequency response, wide dynamic range, low distortion. I’m not interested in colorful components, mics and preamps with a sound, I want the sound to be the sound of my guitar.

      But the last word is that I’m just learning and I hope you find something useful in my posts.