Last week, we took a look at the javax.sound.sampled package and learned how to loop a snippet of audio.

In Part Two, we’re going to take a look at some issues related to audio latency and real-time programming.

This will help us increase the responsiveness of our software, and make for smoother audio transitions.

The High Concept: Reprise

If you’ll recall, we’re working on a program called Looper whose job it is to play a snippet of sound over and over again. It’s also able to switch between different snippets.

Assuming that our snippets are the same length and have the same rhythm, and assuming that we switch properly between them, we should be able to maintain continuity between the different snippets.

Transitioning: The Easy Way

Just switching from one loop to another is easy. Let’s suppose we have two different rhythm tracks which have the same rhythm.

If we want to make a smooth transition from one loop to the other, we can’t simply start playing the second loop from the beginning.

If we did, we would likely find that the second one started right in the middle of the first one, ruining the rhythmic continuity between the snippets and throwing everything off.

But it’s not hard to get this right — all we have to do is keep the value of cursor around. As you remember from Part One, cursor tells us how many bytes of the first loop we’ve written so far:

    // Write at most this many bytes per inner loop execution
    static private final int innerLoopWriteSize = 512;

    // The position within the currently playing sound
    // (i.e. the next sample to write)
    int cursor = 0;

    while (true) {
      // How many bytes are the left to write from this snippet?
      int bytesLeft = snippet.length-cursor;

      // If we've reached the end, start from the top
      // of the sound
      if (bytesLeft<=0) {
        // restart sound
        cursor = 0;
        bytesLeft = currentRaw.length;
      }

      // Don't write more than this in one loop
      int towrite = innerLoopWriteSize;
      if (towrite > bytesLeft)
        towrite = bytesLeft;

      // Write a chunk
      int r = sdl.write( snippet, cursor, towrite );

      // Remember how much we wrote, by advancing the cursor
      // to the next chunk of sound
      cursor += r;
    }

When we switch to the second loop, we need to skip ahead cursor bytes and start writing from that point.

This ensures that the piece of the first snippet matches up with the piece of the second snippet.

Put more concisely, if we’ve played up to sample N of the first snippet, then we start playing the second one starting from sample N.

Following this rule ensures rhythmic continuity. However, we’re going to take it a step further and try to deal with latency.

Latency

The reason that the above solution is not ideal is that, even if we do make a perfect transition, we are not going to hear it right away.

Assuming that the transition is triggered in some way by the user, there will be a delay between this trigger and the moment when the transition becomes audible.

With any digital audio system, it takes a certain amount of time for the audio to get from disk to speaker, or from microphone to disk. (Or from microphone to speaker, for that matter.) This delay is called latency.

For commercial hardware devices, this delay is usually on the order of a few milliseconds or less.

For computer systems, this delay can be substantially longer — sometimes on the order of large fractions of a second.

This increased delay is due to the fact that a desktop workstation is not really designed to provide low-latency audio throughput.

And the audio generally has to go through several levels of software, each of which adds a certain amount of delay.

Compensating For Latency

In our Looper appliation, the effect of the latency is that there is a delay between the time that the user initiates a transition and the time that the transition appears to happen.

It is this delay that we would like to reduce. We would like the transition to occur as soon after initiation as possible.

When using javax.sound.sampled, there are two sources of latency. Any audio leaving your application is first buffered within the code for the javax.sound.sampled library.

It is then sent through the OS audio layers, and then finally to the sound hardware. So we have three potential sources of latency.

In our Java code, there’s nothing we can do about the latency in the OS and hardware layers. The latency in the Java library, however, is under our control to a certain degree.

Let’s consider the moment of transition. At this moment, we’ve written a certain portion of the first snippet to our SourceDataLine, and suddenly the user asks us to transition to the second snippet.

Because of the latency through the system, we know that not all of the audio data that we’ve written has actually gone through.

Some of it may be in the Java buffers, some may be in the OS buffers, and some may be in the hardware buffers.

What we’d really like to do is “recall” the data from these buffers. Sadly, we cannot do this from the OS and hardware buffers, but we can from the Java buffers. Naturally, this is as easy as calling a method:

      sourceDataLine.flush();

What this does is tell the SourceDataLine to empty everything from its buffers. At this point, the SourceDataLine has nothing to play, which means we’d better fill it with new data as fast as possible. Just as we’ve removed the remaining portion of the first snippet, we can start writing the next portion of the second snippet.

A Bit More Complicated

However, we can’t just start playing the second snippet just as we used to do. After all, we just emptied some audio data from the Java buffers, which means we really haven’t played as much audio as we thought we had. The audio we just took out never got played.

We had previously advanced cursor by the amount that we thought we had played.

But we didn’t play all of it, so we need to back up our cursor by an amount equal to the amount of data we flushed from the buffer. The flush() method doesn’t tell us how much was flushed, but we can figure it out for ourselves:

      // We have to flush the buffer, and thus have to
      // back the cursor up a certain amount to keep the
      // looping seamless
      int backlog =
        sourceDataLine.getBufferSize() - sourceDataLine.available();
      cursor -= backlog;

getBufferSize() tells us exactly how large the Java buffer is, while available() tells us how much room is left inside the buffer.

The first minus the second tells us how much data was waiting in the buffer. It is this amount that we need to subtract from the cursor to make sure things line up.

Fudge Factor

In practice, however, even this is not quite enough. The compensation factor above results in a looper that is nearly, but not quite, seamless. A good ear can tell that the transition is not quite … right.

There could be a number of explanations for this, and, frankly, I don’t know for sure which is the right one.

I do know, however, that it is very common for there to be minor errors in latency compensation whenever a general-purpose desktop computer is used for real-time audio.

Even the expensive commercial audio software packages have this problem, and so they make it easy for the user to correct for these small errors by entering a “fudge factor” that is added or subtracted from the expected latency.

Setting the fudge factor properly is mostly a matter of trial-and-error, and a good ear. In our software, we simply hard-code a latency compensation value, and change it until things sound right:

      // the latency through the low-level sound system
      // this must be tuned for each system
      static private final double sysLatencyTime = 0.695; // seconds

      // the sound latency expressed in samples
      static private final int sysLatency =
        (int)(sysLatencyTime*sampRate);

      // ...

      // Back up by a bit more, because of the latency
      // through the low-level sound system
      cursor -= sysLatency*2; // 16 bit, don'tcha know

My system needs a value of 0.695 seconds, and this sounds about right for my sound hardware.

Full Source

Here are links to the full source to a program that demonstrates these concepts. Looper.java contains the looper code. It’s also a main() routine which lets you try it out at the command-line.

To use it, simply enter the filenames of some audio files on the command-line:

prompt> java Looper loop0.wav loop1.wav loop2.wav

Press Return, and Looper will start playing the first sound. Press Return while the program is running, and it will smoothly transition from one sound to the next.

Entering q before pressing Return will cause the program to exit gracefully.

The other class, Queue.java, is a utility class used to hold a list of sounds that are to be played by the Looper.

(In fact, the looper simply skips ahead to the last element in the queue and transitions to that one.)

• Looper.java
• Queue.java

Conclusion

In this two-part series, we’ve learned how to read audio data from the filesystem and play it in a platform- and format- independent way through the sound hardware.

We’ve also learned about some of the more subtle issues surrounding real-time audio, such as latency, and we’ve explored some methods for solving problems associated with these issues.