What is VirtualDub?

VirtualDub is a video capture/processing utility for 32-bit and 64-bit Windows platforms (98/ME/NT4/2000/XP/Vista/7), licensed under the GNU General Public License (GPL).  It lacks the editing power of a general-purpose editor such as Adobe Premiere, but is streamlined for fast linear operations over video.  It has batch-processing capabilities for processing large numbers of files and can be extended with third-party video filters.  VirtualDub is mainly geared toward processing AVI files, although it can read (not write) MPEG-1 and also handle sets of BMP images.

I basically started VirtualDub in college to do some quick capture-and-encoding that I wanted done; from there it's basically grown into a more general utility that can trim and clean up video before exporting to tape or processing with another program.  I released it on the web and others found it useful, so I've been tinkering around with its code ever since.  If you have the time, please download and enjoy.

§ ¶DirectSound Renderer latency woes, part 2

I finally got a little bit of time to took at my live capture audio playback problems again, and I think I have a better idea of what's going on now.

I hacked VirtualDub to display both the buffer status and the tracking rate on the DirectSound Renderer while I played a bit, and what I found was that the renderer was stepping the playback rate up and down at 25Hz at a time, converging on around 49750Hz for the DirectSound playback rate. That's fine, since we have both playback and rendering clocks that are likely off a little bit. What was more disturbing was the buffering level, which had both a random initial level and a very slow upwards climb towards 90%, regardless of where it started. Furthermore, even after it had hit 90%, the renderer still did not change the playback rate, which meant that it was most likely dropping samples periodically at that point in order to keep the buffer level in check.

After starting at the IAMAudioRendererStats entry points and tracing through some portions of the DirectSound Renderer code, I think I have a good idea of how the rate matching algorithm works. The main function is CWaveSlave::AdjustSlaveClock(), which monitors both the master clock (the reference clock set on the renderer) and the slave clock (the DirectSound playback position). The routine monitors the difference between the two and adjusts the playback rate up and down slowly to compensate. I don't know how low it can go, but for a 48KHz sample rate, it clamps out at 48600Hz at the high end. Within these bounds, the renderer matches the rate of the playback clock against the rate of the reference clock.

This sounds good... until you realize the ramifications. In particular, neither buffer level nor data rate are inputs into this algorithm, only clocks. Therefore, the algorithm is only accurate as long as the incoming data rate matches the reference clock. This is fine for file playback, but not such a great assumption for a live source. In my case, there appears to be a discrepancy such that the incoming data rate is actually slightly higher than 48KHz, and therefore the DirectSound Renderer is slightly matching the wrong rate, causing its buffer to fill over time. The other consequence is that the renderer makes no attempt to target any particular latency as long as it isn't encountering either a buffer underflow or overflow condition, which is disturbing.

As it turns out, I inadvertently discovered an amusing workaround, which is to stop the process for a long period of time in the debugger. Upon resuming, this kicks the rate matching algorithm so far off that it clamps out at 48600Hz and then gradually slides into a constant buffer underflow condition. Although the resulting low latency is rather nice, I'm not particularly fond of this "fix."

It looks like the real solution is going to involve me writing a custom filter, either as a transform filter in front of the DirectSound Renderer or just a custom renderer. I'm not sure which one is easier, but I'm actually leaning toward the custom renderer as the random latency issue is a bit bothersome.

§ ¶DirectSound Renderer latency woes

I'm tweaking my laptop-as-a-TV setup again, and am running into problems with latency in the DirectShow filter graph.

The video path I cobbled together a while ago, and works as follows: I have preview acceleration mode activated VirtualDub and set to interlaced fields / even field first, which then pushes the video through a Direct3D pixel shader that does adaptive deinterlacing and upsampling to field rate with a little bit of bicubic filtering and gamma adjustment thrown in. That works surprisingly well, with good latency and a decent image. The problem is the audio, which initially starts out fine but slowly accumulates a lot of latency over time. I've spent a while fiddling with the filter graph clock and perusing the various (many) interfaces, and have determined the following:

• Rate matching by data rate is active on the audio renderer.
• The capture filter is sending 40ms audio buffers, and doesn't seem to respond to IAMBufferNegotiation requests to change it. Therefore, that's 40ms of audio latency I can't get rid of.
• The DirectSound buffer size is about one second.
• The buffer level reported through IAMAudioRendererStats very slowly rises over time, from about 4% to 90%.
• Neither SyncUsingStreamOffset() + SetMaxGraphLatency() nor SetErrorTolerance() appear to help.

I had this same problem with my Adaptec GameBridge, and now it's happening with my ATI capture device, so I don't think it's capture driver specific.

It looks like the current problem is that the DirectSound Renderer is intent upon slowly filling up its one second buffer over time, even though that is a huge amount of latency to add to a live stream. For a live TV stream, that usually isn't a problem, as the audio and video just need to have the same amount of delay to be in sync. However, since I'm trying to play Ar Tonelico, any delay on either the audio or video stream results in a delay relative to the controller inputs, and I really don't want that amount of latency anywhere in the system. It's a bit frustrating that the audio system runs well at first but gradually slews to unusability until I reset the filter graph again.

Does anyone have any experience with this problem or tips as to how to solve it? I really don't want to have to write my own audio renderer, but I'm beginning to think that might be the best solution.