[libavg-devel] On Synching Audio/Video Output

[Ulrich von Zadow]

Nick Hebner wrote:
> Hey All,
> 
> I am starting to think about how to accomplish audio/video synching, so
> I would like to get some input. I have been using the 'How to Write a
> Video Player in Less Than 1000 Lines' tutorial (
> http://www.dranger.com/ffmpeg/) to learn more about ffmpeg and AV
> synching (sections 5 and 6 are relevant to synching). In this tutorial,
> the author presents techniques for defining reference 'clocks' to
> determine how to adjust the streams when they get out of sync. Currently
> I am leaning toward an approach based on synching the video stream to a
> constant audio stream for two reasons:
> 
> 1. From my experience with audio/video data manipulation, it is much
> easier to create audible artifacts than visual ones.

Correct.

> 2. The audio updates happen more quickly than video, meaning that an
> audio clock would be updated more frequently, leaving less error when
> trying to interpolate what its value means.

Right, but at 60 Hz framerate (not necessarily video playback rate), the
maximum error even without interpolation is 17 ms.

The best way I've seen to play back video without artifacts is to sync
the playback to the update of the graphics card and play 30 or 60 Hz
videos. Currently, the 'clock' for video playback is based on the vsync
interval (if we are syncing to vblank) for this reason. The curent
master clock for all of libavg is implemented around Player::m_FrameTime
and DisplayEngine::getDisplayTime(). This way, we can also support
Player::setFakeFPS(). (As an aside, what do we do with audio when
setFakeFPS() is set? Disable it completely? setFakeFPS is meant for
screen recordings and tests which need reproducible results and audio
really shouldn't be essential in this situation.)

On the other hand, for 25 Hz video, syncing to vblank doesn't work
anyway. If you don't produce your videos yourself, this will be most
videos, so I'm not sure what a good solution is. Have an attribute of
the <video> node that chooses whether to use the audio clock to play
back the video? The perfect solution would actually stretch the audio
stream when that becomes necessary, but as you say, that's hard to get
right. And if I remember correctly, the method proposed by the tutorial
you cite sounded wrong when I read it...

> Now that we have support for audio with videos, I think that the 'fps'
> property should become a more general 'speed' property which would
> represent a scalar of both the sample rate and fps. The 'speed' property
> would scale both the sample rate of the audio stream used for resampling
> as well as the fps property for video.

Good idea. Maybe call it 'speedfactor' so it's clear that it's not an
absolute value?

Also, we would need to keep the fps property around for a while (marked
deprecated) or we'll lose backwards compatibility.

Cheers,

  Uli

-- 

Ulrich von Zadow | +49-172-7872715
Jabber: cocacoder at jabber.berlin.ccc.de
Skype: uzadow